2023-01-25T06:17:33.834 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T06:17:33.835 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136911 --name yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136911/orig.config.yaml 2023-01-25T06:17:33.857 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T06:17:33.910 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136911 branch: wip-yuri2-testing-2023-01-23-0928 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} email: yweinstein@radhat.com first_in_suite: false job_id: '7136911' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2023-01-23-0928 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 61996d73ab738798e4ee0055c86457f5c459da4a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 61996d73ab738798e4ee0055c86457f5c459da4a workunit: branch: wip-yuri2-testing-2023-01-23-0928 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2023-01-23-0928 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 61996d73ab738798e4ee0055c86457f5c459da4a targets: smithi040.front.sepia.ceph.com: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw tasks: - install: null - workunit: basedir: qa/standalone clients: all: - mgr teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:22:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T06:17:33.911 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa; will attempt to use it 2023-01-25T06:17:33.911 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa/tasks 2023-01-25T06:17:33.912 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T06:17:33.913 INFO:teuthology.task.internal:Checking packages... 2023-01-25T06:17:33.938 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '61996d73ab738798e4ee0055c86457f5c459da4a' 2023-01-25T06:17:33.939 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T06:17:33.939 INFO:teuthology.packaging:ref: None 2023-01-25T06:17:33.939 INFO:teuthology.packaging:tag: None 2023-01-25T06:17:33.940 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-01-23-0928 2023-01-25T06:17:33.940 INFO:teuthology.packaging:sha1: 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T06:17:33.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2023-01-23-0928 2023-01-25T06:17:34.101 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1963-g61996d73-1focal 2023-01-25T06:17:34.101 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T06:17:34.107 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T06:17:34.107 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T06:17:34.112 INFO:teuthology.task.internal:Saving configuration 2023-01-25T06:17:34.121 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T06:17:34.127 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T06:17:34.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136911', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 06:11:44.449089', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-25T06:17:34.149 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T06:17:34.156 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-25T06:17:34.156 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T06:17:34.190 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2debdc5670>, signals=[15]) 2023-01-25T06:17:34.191 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T06:17:34.201 INFO:teuthology.task.internal:Opening connections... 2023-01-25T06:17:34.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-01-25T06:17:34.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T06:17:34.255 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T06:17:34.262 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-01-25T06:17:34.379 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-01-25T06:17:34.379 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-01-25T06:17:34.385 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2023-01-25T06:17:34.386 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-25T06:17:34.386 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2023-01-25T06:17:34.386 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2023-01-25T06:17:34.386 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-25T06:17:34.387 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="20.04" 2023-01-25T06:17:34.387 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-25T06:17:34.387 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-25T06:17:34.387 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-25T06:17:34.387 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-25T06:17:34.388 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=focal 2023-01-25T06:17:34.388 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=focal 2023-01-25T06:17:34.389 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-01-25T06:17:34.405 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T06:17:34.414 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T06:17:34.420 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T06:17:34.421 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-01-25T06:17:34.427 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T06:17:34.433 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T06:17:34.433 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-01-25T06:17:34.444 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T06:17:34.471 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T06:17:34.485 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T06:17:34.486 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T06:17:34.486 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T06:17:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T06:17:34.492 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-01-25T06:17:34.498 INFO:teuthology.orchestra.run.smithi040.stdout:5.4.0-124-generic 2023-01-25T06:17:34.499 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.4.0-124-generic 2023-01-25T06:17:34.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2023-01-25T06:17:34.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-01-25T06:17:34.798 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T06:17:34.801 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-25T06:17:34.801 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-25T06:17:34.869 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-25T06:17:35.206 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,960 kB] 2023-01-25T06:17:35.341 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [780 kB] 2023-01-25T06:17:35.342 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [543 kB] 2023-01-25T06:17:35.349 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [320 kB] 2023-01-25T06:17:35.353 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-25T06:17:35.354 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-25T06:17:35.354 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,463 kB] 2023-01-25T06:17:35.373 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [207 kB] 2023-01-25T06:17:35.375 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [624 B] 2023-01-25T06:17:35.375 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [787 kB] 2023-01-25T06:17:35.384 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [576 kB] 2023-01-25T06:17:35.390 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-25T06:17:35.392 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-25T06:17:35.407 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,341 kB] 2023-01-25T06:17:35.442 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [404 kB] 2023-01-25T06:17:35.451 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-25T06:17:35.451 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-25T06:17:35.451 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,560 kB] 2023-01-25T06:17:35.459 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [220 kB] 2023-01-25T06:17:35.460 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [620 B] 2023-01-25T06:17:35.461 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [708 kB] 2023-01-25T06:17:35.465 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,019 kB] 2023-01-25T06:17:35.471 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [236 kB] 2023-01-25T06:17:35.473 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.4 kB] 2023-01-25T06:17:35.478 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-25T06:17:35.479 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-25T06:17:35.479 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-25T06:17:35.479 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-25T06:17:35.495 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-25T06:17:35.496 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-25T06:17:39.197 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 13.8 MB in 2s (6,247 kB/s) 2023-01-25T06:17:40.714 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:17:40.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-25T06:17:40.807 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:17:40.999 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-01-25T06:17:41.000 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-01-25T06:17:41.179 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2023-01-25T06:17:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-25T06:17:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-25T06:17:41.180 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-25T06:17:41.181 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2023-01-25T06:17:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-25T06:17:41.181 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2023-01-25T06:17:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: thermald 2023-01-25T06:17:41.250 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2023-01-25T06:17:41.251 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-25T06:17:41.251 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-25T06:17:41.251 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-25T06:17:41.252 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2023-01-25T06:17:41.253 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-25T06:17:41.455 INFO:teuthology.orchestra.run.smithi040.stdout:3 upgraded, 5 newly installed, 0 to remove and 129 not upgraded. 2023-01-25T06:17:41.455 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 77.1 MB of archives. 2023-01-25T06:17:41.455 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-25T06:17:41.456 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-25T06:17:42.714 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-25T06:17:43.013 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-25T06:17:44.406 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-25T06:17:44.406 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-25T06:17:44.407 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-25T06:17:44.711 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-25T06:17:44.769 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-25T06:17:45.476 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 77.1 MB in 4s (22.0 MB/s) 2023-01-25T06:17:45.666 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-25T06:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-25T06:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T06:17:47.480 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:17:49.448 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-25T06:17:49.464 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T06:17:49.564 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:17:50.684 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-25T06:17:50.698 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T06:17:50.759 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:17:55.548 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T06:17:55.683 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T06:17:56.010 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T06:17:56.161 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T06:17:56.523 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-25T06:17:56.539 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-25T06:17:56.581 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-25T06:18:02.684 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-25T06:18:02.705 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-25T06:18:02.767 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:05.563 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-25T06:18:05.692 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-25T06:18:06.038 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-25T06:18:06.154 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:07.448 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:07.574 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:09.167 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-25T06:18:09.167 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-25T06:18:09.310 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:10.528 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-25T06:18:10.679 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-25T06:18:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-25T06:18:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-25T06:18:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-25T06:18:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-25T06:18:33.670 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-25T06:18:33.671 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub' 2023-01-25T06:18:33.695 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-25T06:18:33.714 INFO:teuthology.orchestra.run.smithi040.stdout:Generating grub configuration file ... 2023-01-25T06:18:34.082 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-25T06:18:34.119 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-25T06:18:34.536 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-25T06:18:34.550 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-25T06:18:37.448 INFO:teuthology.orchestra.run.smithi040.stdout:done 2023-01-25T06:18:37.862 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2023-01-25T06:18:37.912 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2023-01-25T06:18:37.913 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2023-01-25T06:18:37.913 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2023-01-25T06:18:37.913 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2023-01-25T06:18:37.914 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 19 2023-01-25T06:18:37.914 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2023-01-25T06:18:37.914 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2023-01-25T06:18:37.914 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2023-01-25T06:18:37.915 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.4.0.137.135 2023-01-25T06:18:37.915 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-25T06:18:37.915 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-25T06:18:37.915 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2023-01-25T06:18:37.915 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2023-01-25T06:18:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2023-01-25T06:18:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2023-01-25T06:18:37.916 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-25T06:18:37.917 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T06:18:37.917 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T06:18:37.917 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T06:18:37.917 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T06:18:37.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2023-01-25T06:18:37.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-01-25T06:18:38.125 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-25T06:18:38.266 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T06:18:38.353 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-25T06:18:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-25T06:18:39.801 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:18:39.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-25T06:18:39.897 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:18:40.075 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-01-25T06:18:40.076 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-01-25T06:18:40.276 INFO:teuthology.orchestra.run.smithi040.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-25T06:18:40.276 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 129 not upgraded. 2023-01-25T06:18:40.278 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2023-01-25T06:18:40.301 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2023-01-25T06:18:40.301 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2023-01-25T06:18:40.301 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2023-01-25T06:18:40.301 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2023-01-25T06:18:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 19 2023-01-25T06:18:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2023-01-25T06:18:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2023-01-25T06:18:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2023-01-25T06:18:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.4.0.137.135 2023-01-25T06:18:40.303 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-25T06:18:40.303 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-25T06:18:40.303 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2023-01-25T06:18:40.303 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2023-01-25T06:18:40.303 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2023-01-25T06:18:40.304 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2023-01-25T06:18:40.304 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-01-25T06:18:40.311 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.exjs0iBWiQ 2023-01-25T06:18:40.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub/grub.cfg /tmp/tmp.exjs0iBWiQ 2023-01-25T06:18:40.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.exjs0iBWiQ 2023-01-25T06:18:40.407 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.exjs0iBWiQ is 9KB 2023-01-25T06:18:40.422 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.exjs0iBWiQ 2023-01-25T06:18:40.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-25T06:18:40.484 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:18:40.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-25T06:18:40.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-25T06:18:40.513 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-25T06:18:40.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo update-grub 2023-01-25T06:18:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2023-01-25T06:18:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-25T06:18:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2023-01-25T06:18:42.286 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-25T06:18:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-25T06:18:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-25T06:18:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-25T06:18:44.319 INFO:teuthology.orchestra.run.smithi040.stderr:done 2023-01-25T06:18:44.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2023-01-25T06:19:14.352 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T06:19:14.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-01-25T06:19:14.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T06:19:32.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-01-25T06:20:02.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-01-25T06:20:02.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T06:20:18.544 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-01-25T06:20:19.376 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-01-25T06:20:19.376 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-137-generic"... 2023-01-25T06:20:19.377 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-01-25T06:20:19.383 INFO:teuthology.orchestra.run.smithi040.stdout:5.4.0-137-generic 2023-01-25T06:20:19.383 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-25T06:20:19.384 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T06:20:19.384 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-25T06:20:20.385 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T06:20:20.386 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T06:20:20.475 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2023-01-25T06:20:20.499 DEBUG:teuthology.parallel:result is None 2023-01-25T06:20:20.500 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T06:20:20.507 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T06:20:20.508 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T06:20:20.515 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T06:20:20.521 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T06:20:20.531 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T06:20:20.531 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T06:20:20.589 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T06:20:20.596 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T06:20:20.597 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T06:20:20.615 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:20:20.626 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:20:20.628 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T06:20:20.635 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T06:20:20.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T06:20:20.650 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T06:20:20.661 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T06:20:20.661 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T06:20:20.699 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T06:20:20.707 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T06:20:20.715 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:20:20.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T06:20:20.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-01-25T06:20:20.775 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T06:20:20.783 INFO:teuthology.task.internal:Starting timer... 2023-01-25T06:20:20.783 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T06:20:20.793 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T06:20:20.804 DEBUG:teuthology.task.selinux:Excluding smithi040: OS 'ubuntu' does not support SELinux 2023-01-25T06:20:20.804 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T06:20:20.805 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-01-25T06:20:20.805 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T06:20:20.805 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T06:20:20.814 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-25T06:20:20.815 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T06:20:20.830 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T06:20:20.831 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T06:25:01.100 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com')] 2023-01-25T06:25:01.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-01-25T06:25:01.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T06:25:01.151 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-01-25T06:25:01.282 INFO:teuthology.run_tasks:Running task clock... 2023-01-25T06:25:01.293 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-25T06:25:01.293 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T06:25:01.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T06:25:01.328 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-25T06:25:01.329 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Command line: ntpd -gq 2023-01-25T06:25:01.330 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: proto: precision = 0.065 usec (-24) 2023-01-25T06:25:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-25T06:25:01.331 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-25T06:25:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-25T06:25:01.332 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: restrict ::: KOD does nothing without LIMITED. 2023-01-25T06:25:01.332 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen and drop on 0 v6wildcard [::]:123 2023-01-25T06:25:01.333 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-25T06:25:01.333 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen normally on 2 lo 127.0.0.1:123 2023-01-25T06:25:01.333 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen normally on 3 ens1f0 172.21.15.40:123 2023-01-25T06:25:01.333 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen normally on 4 lo [::1]:123 2023-01-25T06:25:01.334 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12cc%3]:123 2023-01-25T06:25:01.334 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:01 ntpd[19361]: Listening on routing socket on fd #22 for interface updates 2023-01-25T06:25:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:03 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:04.329 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:04.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:04.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:04.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:04.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:04.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:04 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:05 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:06.329 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:06.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:06 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:07.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:07.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:07.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:07.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:07.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:07.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:07.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:07.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:07.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:07.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:07.332 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:07.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:07 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:08.329 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:08.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:08.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:08.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:08.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:08.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:08 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:09.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:09.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:09.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:09.330 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-25T06:25:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: can't open /var/log/ntpstats/rawstats.20230125: Permission denied 2023-01-25T06:25:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-25T06:25:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: can't open /var/log/ntpstats/peerstats.20230125: Permission denied 2023-01-25T06:25:09.332 INFO:teuthology.orchestra.run.smithi040.stdout:25 Jan 06:25:09 ntpd[19361]: ntpd: time slew +0.005424 s 2023-01-25T06:25:09.332 INFO:teuthology.orchestra.run.smithi040.stdout:ntpd: time slew +0.005424s 2023-01-25T06:25:09.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-25T06:25:09.333 INFO:teuthology.orchestra.run.smithi040.stderr:25 Jan 06:25:09 ntpd[19361]: can't open /var/log/ntpstats/loopstats.20230125: Permission denied 2023-01-25T06:25:09.395 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2023-01-25T06:25:09.395 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2023-01-25T06:25:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T06:25:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T06:25:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T06:25:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-25T06:25:09.397 INFO:teuthology.run_tasks:Running task install... 2023-01-25T06:25:09.409 DEBUG:teuthology.task.install:project ceph 2023-01-25T06:25:09.409 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}} 2023-01-25T06:25:09.410 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'} 2023-01-25T06:25:09.410 INFO:teuthology.task.install:Using flavor: default 2023-01-25T06:25:09.417 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-25T06:25:09.418 INFO:teuthology.task.install:extra packages: [] 2023-01-25T06:25:09.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2023-01-25T06:25:09.468 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2023-01-25T06:25:09.473 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T06:25:09.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-key list | grep Ceph 2023-01-25T06:25:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-25T06:25:09.637 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-25T06:25:09.638 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph.com (release key) 2023-01-25T06:25:09.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2023-01-25T06:25:09.689 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2023-01-25T06:25:09.694 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T06:25:09.695 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-25T06:25:09.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T06:25:09.848 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default/ 2023-01-25T06:25:09.849 INFO:teuthology.task.install.deb:Package version is 18.0.0-1963-g61996d73-1focal 2023-01-25T06:25:09.966 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:25:09.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-25T06:25:09.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-01-25T06:25:10.232 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal InRelease 2023-01-25T06:25:10.255 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-25T06:25:10.280 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal Release.gpg 2023-01-25T06:25:10.304 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main i386 Packages [3,428 B] 2023-01-25T06:25:10.349 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-25T06:25:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-25T06:25:11.380 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:7 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-25T06:25:11.403 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-25T06:25:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-25T06:25:11.920 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 28.4 kB in 2s (16.5 kB/s) 2023-01-25T06:25:12.895 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:25:12.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-1963-g61996d73-1focal cephadm=18.0.0-1963-g61996d73-1focal ceph-mds=18.0.0-1963-g61996d73-1focal ceph-mgr=18.0.0-1963-g61996d73-1focal ceph-common=18.0.0-1963-g61996d73-1focal ceph-fuse=18.0.0-1963-g61996d73-1focal ceph-test=18.0.0-1963-g61996d73-1focal ceph-volume=18.0.0-1963-g61996d73-1focal radosgw=18.0.0-1963-g61996d73-1focal python3-rados=18.0.0-1963-g61996d73-1focal python3-rgw=18.0.0-1963-g61996d73-1focal python3-cephfs=18.0.0-1963-g61996d73-1focal python3-rbd=18.0.0-1963-g61996d73-1focal libcephfs2=18.0.0-1963-g61996d73-1focal libcephfs-dev=18.0.0-1963-g61996d73-1focal librados2=18.0.0-1963-g61996d73-1focal librbd1=18.0.0-1963-g61996d73-1focal rbd-fuse=18.0.0-1963-g61996d73-1focal 2023-01-25T06:25:12.990 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-01-25T06:25:13.182 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-01-25T06:25:13.182 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-01-25T06:25:13.358 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2023-01-25T06:25:13.360 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-25T06:25:13.360 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-25T06:25:13.360 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2023-01-25T06:25:13.360 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-25T06:25:13.361 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-25T06:25:13.361 INFO:teuthology.orchestra.run.smithi040.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-25T06:25:13.361 INFO:teuthology.orchestra.run.smithi040.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-25T06:25:13.361 INFO:teuthology.orchestra.run.smithi040.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-25T06:25:13.362 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-25T06:25:13.362 INFO:teuthology.orchestra.run.smithi040.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-01-25T06:25:13.362 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-01-25T06:25:13.363 INFO:teuthology.orchestra.run.smithi040.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-25T06:25:13.363 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-01-25T06:25:13.363 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-01-25T06:25:13.363 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-25T06:25:13.363 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-25T06:25:13.364 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-25T06:25:13.364 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-25T06:25:13.364 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-25T06:25:13.364 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-01-25T06:25:13.364 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-01-25T06:25:13.365 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-25T06:25:13.365 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-25T06:25:13.365 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-25T06:25:13.365 INFO:teuthology.orchestra.run.smithi040.stdout: qttranslations5-l10n socat xmlstarlet 2023-01-25T06:25:13.365 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2023-01-25T06:25:13.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-25T06:25:13.366 INFO:teuthology.orchestra.run.smithi040.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-25T06:25:13.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-25T06:25:13.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-25T06:25:13.366 INFO:teuthology.orchestra.run.smithi040.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-25T06:25:13.367 INFO:teuthology.orchestra.run.smithi040.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-25T06:25:13.367 INFO:teuthology.orchestra.run.smithi040.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-25T06:25:13.367 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-25T06:25:13.367 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-25T06:25:13.367 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2023-01-25T06:25:13.368 INFO:teuthology.orchestra.run.smithi040.stdout: btrfs-tools 2023-01-25T06:25:13.478 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2023-01-25T06:25:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-25T06:25:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-25T06:25:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-25T06:25:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-25T06:25:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-25T06:25:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-25T06:25:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-25T06:25:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-01-25T06:25:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-25T06:25:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-25T06:25:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-25T06:25:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-25T06:25:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-25T06:25:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-01-25T06:25:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-01-25T06:25:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-01-25T06:25:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-01-25T06:25:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-01-25T06:25:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-01-25T06:25:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet 2023-01-25T06:25:13.485 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2023-01-25T06:25:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 librbd1 2023-01-25T06:25:13.637 INFO:teuthology.orchestra.run.smithi040.stdout:2 upgraded, 110 newly installed, 0 to remove and 124 not upgraded. 2023-01-25T06:25:13.637 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 210 MB of archives. 2023-01-25T06:25:13.638 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 1,029 MB of additional disk space will be used. 2023-01-25T06:25:13.638 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.0.0-1963-g61996d73-1focal [3,428 kB] 2023-01-25T06:25:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-25T06:25:13.849 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.0.0-1963-g61996d73-1focal [3,963 kB] 2023-01-25T06:25:13.971 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.0.0-1963-g61996d73-1focal [719 kB] 2023-01-25T06:25:13.994 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.0.0-1963-g61996d73-1focal [376 kB] 2023-01-25T06:25:14.006 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.0.0-1963-g61996d73-1focal [22.4 kB] 2023-01-25T06:25:14.007 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.0.0-1963-g61996d73-1focal [184 kB] 2023-01-25T06:25:14.013 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.0.0-1963-g61996d73-1focal [46.5 kB] 2023-01-25T06:25:14.014 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.0.0-1963-g61996d73-1focal [354 kB] 2023-01-25T06:25:14.026 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.0.0-1963-g61996d73-1focal [6,516 kB] 2023-01-25T06:25:14.240 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.0.0-1963-g61996d73-1focal [104 kB] 2023-01-25T06:25:14.243 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.0.0-1963-g61996d73-1focal [469 kB] 2023-01-25T06:25:14.258 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.0.0-1963-g61996d73-1focal [29.6 MB] 2023-01-25T06:25:14.363 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-25T06:25:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-25T06:25:15.373 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.0.0-1963-g61996d73-1focal [10.7 MB] 2023-01-25T06:25:15.609 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.0.0-1963-g61996d73-1focal [221 kB] 2023-01-25T06:25:15.616 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.0.0-1963-g61996d73-1focal [133 kB] 2023-01-25T06:25:15.621 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.0.0-1963-g61996d73-1focal [1,533 kB] 2023-01-25T06:25:15.671 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.0.0-1963-g61996d73-1focal [7,857 kB] 2023-01-25T06:25:15.929 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.0.0-1963-g61996d73-1focal [28.9 MB] 2023-01-25T06:25:16.531 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-25T06:25:16.964 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.0.0-1963-g61996d73-1focal [3,808 B] 2023-01-25T06:25:17.036 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.0.0-1963-g61996d73-1focal [829 kB] 2023-01-25T06:25:17.049 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.0.0-1963-g61996d73-1focal [2,279 kB] 2023-01-25T06:25:17.066 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.0.0-1963-g61996d73-1focal [96.2 kB] 2023-01-25T06:25:17.066 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.0.0-1963-g61996d73-1focal [115 kB] 2023-01-25T06:25:17.067 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.0.0-1963-g61996d73-1focal [1,787 kB] 2023-01-25T06:25:17.085 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.0.0-1963-g61996d73-1focal [7,891 kB] 2023-01-25T06:25:17.345 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.0.0-1963-g61996d73-1focal [3,928 B] 2023-01-25T06:25:17.345 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.0.0-1963-g61996d73-1focal [53.4 MB] 2023-01-25T06:25:19.749 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.0.0-1963-g61996d73-1focal [225 kB] 2023-01-25T06:25:19.753 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.0.0-1963-g61996d73-1focal [27.4 kB] 2023-01-25T06:25:19.754 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.0.0-1963-g61996d73-1focal [13.3 MB] 2023-01-25T06:25:20.322 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.0.0-1963-g61996d73-1focal [97.2 kB] 2023-01-25T06:25:22.454 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-25T06:25:22.591 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-25T06:25:22.743 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-25T06:25:22.831 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-25T06:25:22.879 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-25T06:25:22.939 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-25T06:25:22.958 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-25T06:25:23.064 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-25T06:25:23.266 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-25T06:25:24.040 INFO:teuthology.orchestra.run.smithi040.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-25T06:25:24.112 INFO:teuthology.orchestra.run.smithi040.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-25T06:25:24.158 INFO:teuthology.orchestra.run.smithi040.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-25T06:25:24.202 INFO:teuthology.orchestra.run.smithi040.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-25T06:25:24.309 INFO:teuthology.orchestra.run.smithi040.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-25T06:25:24.342 INFO:teuthology.orchestra.run.smithi040.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-25T06:25:24.365 INFO:teuthology.orchestra.run.smithi040.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-25T06:25:24.469 INFO:teuthology.orchestra.run.smithi040.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-25T06:25:24.515 INFO:teuthology.orchestra.run.smithi040.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-25T06:25:24.594 INFO:teuthology.orchestra.run.smithi040.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-25T06:25:25.346 INFO:teuthology.orchestra.run.smithi040.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-25T06:25:25.409 INFO:teuthology.orchestra.run.smithi040.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-25T06:25:27.729 INFO:teuthology.orchestra.run.smithi040.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-25T06:25:27.754 INFO:teuthology.orchestra.run.smithi040.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-25T06:25:27.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-25T06:25:28.048 INFO:teuthology.orchestra.run.smithi040.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-25T06:25:28.214 INFO:teuthology.orchestra.run.smithi040.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-25T06:25:28.384 INFO:teuthology.orchestra.run.smithi040.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-25T06:25:28.494 INFO:teuthology.orchestra.run.smithi040.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-25T06:25:28.526 INFO:teuthology.orchestra.run.smithi040.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-01-25T06:25:28.589 INFO:teuthology.orchestra.run.smithi040.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-25T06:25:28.700 INFO:teuthology.orchestra.run.smithi040.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-25T06:25:28.739 INFO:teuthology.orchestra.run.smithi040.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-25T06:25:28.761 INFO:teuthology.orchestra.run.smithi040.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-25T06:25:28.789 INFO:teuthology.orchestra.run.smithi040.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-25T06:25:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-25T06:25:29.074 INFO:teuthology.orchestra.run.smithi040.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-25T06:25:29.105 INFO:teuthology.orchestra.run.smithi040.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-25T06:25:29.174 INFO:teuthology.orchestra.run.smithi040.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-25T06:25:29.273 INFO:teuthology.orchestra.run.smithi040.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-25T06:25:30.750 INFO:teuthology.orchestra.run.smithi040.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-25T06:25:30.830 INFO:teuthology.orchestra.run.smithi040.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-25T06:25:31.128 INFO:teuthology.orchestra.run.smithi040.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-25T06:25:31.282 INFO:teuthology.orchestra.run.smithi040.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-25T06:25:31.377 INFO:teuthology.orchestra.run.smithi040.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-25T06:25:31.401 INFO:teuthology.orchestra.run.smithi040.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-25T06:25:31.475 INFO:teuthology.orchestra.run.smithi040.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-25T06:25:37.787 INFO:teuthology.orchestra.run.smithi040.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-25T06:25:42.123 INFO:teuthology.orchestra.run.smithi040.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-25T06:25:42.175 INFO:teuthology.orchestra.run.smithi040.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-25T06:25:42.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-25T06:25:42.372 INFO:teuthology.orchestra.run.smithi040.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-25T06:25:42.391 INFO:teuthology.orchestra.run.smithi040.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-25T06:25:42.410 INFO:teuthology.orchestra.run.smithi040.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-25T06:25:42.428 INFO:teuthology.orchestra.run.smithi040.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-25T06:25:42.447 INFO:teuthology.orchestra.run.smithi040.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-25T06:25:42.465 INFO:teuthology.orchestra.run.smithi040.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-25T06:25:42.486 INFO:teuthology.orchestra.run.smithi040.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-25T06:25:42.505 INFO:teuthology.orchestra.run.smithi040.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-25T06:25:42.523 INFO:teuthology.orchestra.run.smithi040.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-25T06:25:42.555 INFO:teuthology.orchestra.run.smithi040.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-25T06:25:42.580 INFO:teuthology.orchestra.run.smithi040.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-25T06:25:42.604 INFO:teuthology.orchestra.run.smithi040.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-25T06:25:42.623 INFO:teuthology.orchestra.run.smithi040.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-25T06:25:42.658 INFO:teuthology.orchestra.run.smithi040.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-25T06:25:42.690 INFO:teuthology.orchestra.run.smithi040.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-25T06:25:48.651 INFO:teuthology.orchestra.run.smithi040.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-25T06:25:49.175 INFO:teuthology.orchestra.run.smithi040.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-01-25T06:25:49.329 INFO:teuthology.orchestra.run.smithi040.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-25T06:25:49.821 INFO:teuthology.orchestra.run.smithi040.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-25T06:25:49.914 INFO:teuthology.orchestra.run.smithi040.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-25T06:25:50.006 INFO:teuthology.orchestra.run.smithi040.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-25T06:25:50.100 INFO:teuthology.orchestra.run.smithi040.stdout:Get:107 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-25T06:25:50.213 INFO:teuthology.orchestra.run.smithi040.stdout:Get:108 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-25T06:25:50.306 INFO:teuthology.orchestra.run.smithi040.stdout:Get:109 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-25T06:25:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:Get:110 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-25T06:25:50.493 INFO:teuthology.orchestra.run.smithi040.stdout:Get:111 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-25T06:25:50.586 INFO:teuthology.orchestra.run.smithi040.stdout:Get:112 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-25T06:25:51.251 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 210 MB in 37s (5,671 kB/s) 2023-01-25T06:25:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-25T06:25:51.476 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-25T06:25:51.480 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-25T06:25:51.550 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-25T06:25:51.870 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-25T06:25:51.887 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-25T06:25:51.929 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-25T06:25:52.265 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-25T06:25:52.282 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-25T06:25:52.332 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-25T06:25:52.785 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-25T06:25:52.802 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T06:25:52.854 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:25:53.355 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-25T06:25:53.371 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T06:25:53.413 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:25:53.791 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-25T06:25:53.807 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-25T06:25:53.849 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:25:54.269 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-25T06:25:54.287 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-25T06:25:54.327 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-25T06:25:54.723 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../007-librbd1_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:54.847 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librbd1 (18.0.0-1963-g61996d73-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-25T06:25:55.535 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../008-librados2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:55.694 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librados2 (18.0.0-1963-g61996d73-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-25T06:25:56.439 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs2. 2023-01-25T06:25:56.455 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:56.497 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:56.859 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rados. 2023-01-25T06:25:56.875 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../010-python3-rados_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:56.917 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rados (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:57.219 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-25T06:25:57.236 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:25:57.269 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-argparse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:57.522 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cephfs. 2023-01-25T06:25:57.539 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:57.580 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cephfs (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:57.908 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-25T06:25:57.924 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:25:57.991 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:58.292 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-prettytable. 2023-01-25T06:25:58.308 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-01-25T06:25:58.350 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-25T06:25:58.619 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rbd. 2023-01-25T06:25:58.635 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../015-python3-rbd_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:25:58.677 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rbd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:25:58.988 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-25T06:25:59.006 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-25T06:25:59.071 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-25T06:25:59.366 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-25T06:25:59.385 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-25T06:25:59.424 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-25T06:25:59.786 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-25T06:25:59.802 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-25T06:25:59.844 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-25T06:26:00.181 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librgw2. 2023-01-25T06:26:00.198 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../019-librgw2_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:00.239 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librgw2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:01.484 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rgw. 2023-01-25T06:26:01.500 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:01.541 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:01.920 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liboath0. 2023-01-25T06:26:01.936 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-25T06:26:01.978 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-25T06:26:02.289 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libradosstriper1. 2023-01-25T06:26:02.306 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:02.372 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libradosstriper1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:02.750 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-common. 2023-01-25T06:26:02.769 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../023-ceph-common_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:02.800 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:05.664 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-base. 2023-01-25T06:26:05.683 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../024-ceph-base_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:05.724 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-base (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:07.193 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-25T06:26:07.210 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-25T06:26:07.268 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-25T06:26:07.630 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-natsort. 2023-01-25T06:26:07.649 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-01-25T06:26:07.688 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-25T06:26:08.025 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-logutils. 2023-01-25T06:26:08.041 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-01-25T06:26:08.083 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-25T06:26:08.385 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-25T06:26:08.402 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-25T06:26:08.468 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-25T06:26:08.738 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-mako. 2023-01-25T06:26:08.757 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-25T06:26:08.788 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-25T06:26:09.141 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-25T06:26:09.159 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-25T06:26:09.199 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-25T06:26:09.511 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-25T06:26:09.528 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-25T06:26:09.594 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-25T06:26:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webob. 2023-01-25T06:26:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-25T06:26:09.955 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-25T06:26:10.311 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-25T06:26:10.327 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-25T06:26:10.369 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-25T06:26:10.696 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-bs4. 2023-01-25T06:26:10.716 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-01-25T06:26:10.763 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-25T06:26:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-waitress. 2023-01-25T06:26:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-25T06:26:11.195 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-25T06:26:11.561 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempita. 2023-01-25T06:26:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-01-25T06:26:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-25T06:26:11.948 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-paste. 2023-01-25T06:26:11.966 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-25T06:26:12.005 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-25T06:26:12.351 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-25T06:26:12.369 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-25T06:26:12.409 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-25T06:26:12.712 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-25T06:26:12.728 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-25T06:26:12.770 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-25T06:26:13.081 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webtest. 2023-01-25T06:26:13.101 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-01-25T06:26:13.139 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-25T06:26:13.492 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pecan. 2023-01-25T06:26:13.512 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-25T06:26:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-25T06:26:13.897 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-25T06:26:13.917 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-25T06:26:13.946 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-25T06:26:14.267 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-25T06:26:14.283 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:14.325 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-modules-core (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:14.713 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-25T06:26:14.733 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:14.771 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:15.099 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cryptography. 2023-01-25T06:26:15.115 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-25T06:26:15.173 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-25T06:26:15.544 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-openssl. 2023-01-25T06:26:15.562 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-25T06:26:15.602 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-25T06:26:15.980 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr. 2023-01-25T06:26:15.998 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../047-ceph-mgr_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:16.039 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:16.552 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mon. 2023-01-25T06:26:16.572 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../048-ceph-mon_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:16.610 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mon (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:17.781 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-osd. 2023-01-25T06:26:17.801 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../049-ceph-osd_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:17.830 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-osd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:20.800 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph. 2023-01-25T06:26:20.816 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../050-ceph_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:20.858 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:21.178 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-fuse. 2023-01-25T06:26:21.195 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../051-ceph-fuse_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:21.261 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mds. 2023-01-25T06:26:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../052-ceph-mds_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mds (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:22.218 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephadm. 2023-01-25T06:26:22.234 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../053-cephadm_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:22.276 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:22.604 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-asyncssh. 2023-01-25T06:26:22.622 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-01-25T06:26:22.662 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-01-25T06:26:23.049 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jinja2. 2023-01-25T06:26:23.065 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-01-25T06:26:23.107 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-25T06:26:23.443 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-25T06:26:23.459 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:23.501 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:23.829 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-25T06:26:23.849 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-25T06:26:23.887 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-25T06:26:24.232 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jwt. 2023-01-25T06:26:24.248 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-25T06:26:24.290 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-25T06:26:24.635 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-attr. 2023-01-25T06:26:24.651 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-01-25T06:26:24.701 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-25T06:26:25.066 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pluggy. 2023-01-25T06:26:25.085 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-01-25T06:26:25.129 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-25T06:26:25.449 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-py. 2023-01-25T06:26:25.469 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-25T06:26:25.498 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-25T06:26:25.860 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-25T06:26:25.880 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-25T06:26:25.927 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-25T06:26:26.255 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-packaging. 2023-01-25T06:26:26.271 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-01-25T06:26:26.313 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-25T06:26:26.615 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-25T06:26:26.632 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-25T06:26:26.673 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-25T06:26:27.001 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pytest. 2023-01-25T06:26:27.019 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-01-25T06:26:27.059 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-25T06:26:27.422 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-25T06:26:27.439 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-01-25T06:26:27.480 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-25T06:26:27.716 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjs-underscore. 2023-01-25T06:26:27.734 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-25T06:26:27.773 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-25T06:26:28.060 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-25T06:26:28.078 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-25T06:26:28.117 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-25T06:26:28.479 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-routes. 2023-01-25T06:26:28.496 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-01-25T06:26:28.537 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-25T06:26:28.849 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-25T06:26:28.866 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:28.923 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-dashboard (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:29.416 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-decorator. 2023-01-25T06:26:29.435 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-25T06:26:29.482 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-25T06:26:29.927 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-25T06:26:29.943 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-25T06:26:29.985 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-25T06:26:30.338 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-scipy. 2023-01-25T06:26:30.355 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-25T06:26:30.396 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-25T06:26:31.682 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-25T06:26:31.699 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-25T06:26:31.756 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-25T06:26:32.554 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-joblib. 2023-01-25T06:26:32.573 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-01-25T06:26:32.612 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-25T06:26:33.025 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn. 2023-01-25T06:26:33.041 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-25T06:26:33.083 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-25T06:26:33.759 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-25T06:26:33.776 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:33.816 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:34.970 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cachetools. 2023-01-25T06:26:34.986 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-01-25T06:26:35.028 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-25T06:26:35.364 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-25T06:26:35.380 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-25T06:26:35.422 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-25T06:26:35.759 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-25T06:26:35.778 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-25T06:26:35.816 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-25T06:26:36.231 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rsa. 2023-01-25T06:26:36.250 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-01-25T06:26:36.288 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-25T06:26:36.601 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-google-auth. 2023-01-25T06:26:36.617 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-01-25T06:26:36.659 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-25T06:26:37.013 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-blinker. 2023-01-25T06:26:37.031 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-25T06:26:37.071 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-25T06:26:37.374 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-25T06:26:37.391 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-25T06:26:37.432 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-25T06:26:37.728 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-25T06:26:37.746 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-25T06:26:37.787 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-25T06:26:38.115 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-websocket. 2023-01-25T06:26:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-25T06:26:38.174 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-25T06:26:38.511 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-25T06:26:38.528 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-25T06:26:38.569 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-25T06:26:39.197 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-25T06:26:39.215 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:39.247 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:39.616 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-25T06:26:39.635 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-01-25T06:26:39.675 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-25T06:26:40.062 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-25T06:26:40.081 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-25T06:26:40.120 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-25T06:26:40.389 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package jq. 2023-01-25T06:26:40.406 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-25T06:26:40.439 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-25T06:26:40.684 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package socat. 2023-01-25T06:26:40.702 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-01-25T06:26:40.741 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-25T06:26:41.054 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package xmlstarlet. 2023-01-25T06:26:41.071 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-25T06:26:41.120 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-25T06:26:41.475 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-test. 2023-01-25T06:26:41.492 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../094-ceph-test_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:41.533 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-test (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:46.345 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-volume. 2023-01-25T06:26:46.365 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../095-ceph-volume_18.0.0-1963-g61996d73-1focal_all.deb ... 2023-01-25T06:26:46.411 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-volume (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:46.750 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-25T06:26:46.767 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-25T06:26:46.808 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-25T06:26:47.353 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-25T06:26:47.373 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../097-libcephfs-dev_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:47.411 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs-dev (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:47.832 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package nvme-cli. 2023-01-25T06:26:47.849 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-25T06:26:47.908 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-25T06:26:48.245 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-01-25T06:26:48.263 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-01-25T06:26:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-01-25T06:26:48.699 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-25T06:26:48.715 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-25T06:26:48.756 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-25T06:26:50.456 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-25T06:26:50.475 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-25T06:26:50.512 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-25T06:26:50.866 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webencodings. 2023-01-25T06:26:50.886 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-25T06:26:50.924 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-25T06:26:51.252 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-html5lib. 2023-01-25T06:26:51.269 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-01-25T06:26:51.309 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-25T06:26:51.646 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-25T06:26:51.664 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-25T06:26:51.704 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-25T06:26:52.133 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastescript. 2023-01-25T06:26:52.151 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-01-25T06:26:52.190 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-25T06:26:52.561 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-psutil. 2023-01-25T06:26:52.578 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-25T06:26:52.619 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-25T06:26:52.989 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-25T06:26:53.010 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-25T06:26:53.047 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-25T06:26:53.324 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplejson. 2023-01-25T06:26:53.343 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-25T06:26:53.374 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-25T06:26:53.627 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-25T06:26:53.644 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-25T06:26:53.685 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-25T06:26:54.406 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package radosgw. 2023-01-25T06:26:54.424 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../110-radosgw_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:54.455 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking radosgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package rbd-fuse. 2023-01-25T06:26:55.895 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../111-rbd-fuse_18.0.0-1963-g61996d73-1focal_amd64.deb ... 2023-01-25T06:26:55.933 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking rbd-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:56.235 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-25T06:26:56.538 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-25T06:26:56.664 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-25T06:26:56.815 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-25T06:26:57.008 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:26:57.628 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user cephadm....done 2023-01-25T06:26:57.756 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-25T06:26:57.915 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-25T06:26:58.284 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-25T06:26:58.577 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-25T06:26:58.694 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-25T06:26:58.820 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-25T06:26:59.139 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-25T06:26:59.566 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-25T06:26:59.859 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-25T06:27:00.144 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-25T06:27:00.419 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-25T06:27:00.712 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-argparse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:00.997 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-25T06:27:01.122 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-25T06:27:01.423 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-25T06:27:01.741 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-25T06:27:01.867 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-25T06:27:02.244 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-25T06:27:02.536 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-25T06:27:03.104 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-25T06:27:03.455 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-25T06:27:03.765 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-25T06:27:04.107 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-25T06:27:04.400 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-25T06:27:04.701 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-25T06:27:04.811 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-25T06:27:04.928 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-25T06:27:05.221 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-25T06:27:05.606 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-25T06:27:05.991 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-25T06:27:06.368 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-25T06:27:06.494 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:27:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-25T06:27:07.339 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-25T06:27:07.481 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-25T06:27:07.616 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-25T06:27:07.917 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-25T06:27:08.235 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:27:08.344 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-25T06:27:08.504 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-25T06:27:08.830 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-25T06:27:09.123 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-25T06:27:09.424 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-25T06:27:09.575 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-25T06:27:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-25T06:27:10.236 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-25T06:27:10.528 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-25T06:27:10.740 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-25T06:27:10.988 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-25T06:27:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-25T06:27:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-25T06:27:11.834 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-25T06:27:11.960 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-25T06:27:12.102 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-25T06:27:12.428 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-25T06:27:12.713 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-25T06:27:14.227 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-25T06:27:14.344 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-25T06:27:14.704 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-25T06:27:14.997 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-25T06:27:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-25T06:27:15.441 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-25T06:27:15.809 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-25T06:27:16.144 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-01-25T06:27:16.512 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-25T06:27:16.838 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-25T06:27:17.933 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-25T06:27:18.059 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-25T06:27:18.185 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-25T06:27:18.336 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-25T06:27:18.654 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-25T06:27:18.780 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-25T06:27:19.106 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-25T06:27:19.424 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-25T06:27:19.784 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-25T06:27:20.144 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librados2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:20.253 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-01-25T06:27:20.380 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librgw2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:20.539 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libsqlite3-mod-ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:20.751 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs2 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:20.877 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-25T06:27:21.178 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libradosstriper1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:21.287 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-25T06:27:21.602 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librbd1 (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:21.753 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-modules-core (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:21.862 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:22.074 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-25T06:27:22.074 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-25T06:27:22.555 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs-dev (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:22.682 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rados (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:22.848 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-25T06:27:23.819 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rbd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:23.978 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up rbd-fuse (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:24.104 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:24.230 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cephfs (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:24.356 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-common (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:Adding group ceph....done 2023-01-25T06:27:25.319 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user ceph....done 2023-01-25T06:27:25.496 INFO:teuthology.orchestra.run.smithi040.stdout:Setting system user ceph properties....done 2023-01-25T06:27:25.517 INFO:teuthology.orchestra.run.smithi040.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-25T06:27:25.669 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-25T06:27:26.001 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-25T06:27:26.442 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-test (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:26.568 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up radosgw (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-25T06:27:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-25T06:27:27.553 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-base (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:27.962 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-25T06:27:28.422 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mds (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:28.780 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-25T06:27:28.780 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-25T06:27:29.215 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:29.434 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-25T06:27:29.434 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-25T06:27:29.883 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-osd (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:30.413 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-25T06:27:30.414 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-25T06:27:30.860 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-k8sevents (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:30.986 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:31.154 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mon (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:31.385 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-25T06:27:31.385 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-25T06:27:31.847 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-cephadm (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:31.989 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:32.115 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-dashboard (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:32.283 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-volume (18.0.0-1963-g61996d73-1focal) ... 2023-01-25T06:27:32.394 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-25T06:27:32.537 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-25T06:27:32.822 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-25T06:27:35.841 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-25T06:27:36.415 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-25T06:27:36.418 DEBUG:teuthology.parallel:result is None 2023-01-25T06:27:36.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2023-01-25T06:27:36.467 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2023-01-25T06:27:36.472 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-25T06:27:36.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T06:27:36.635 DEBUG:teuthology.orchestra.run.smithi040:> dpkg-query -W -f '${Version}' ceph 2023-01-25T06:27:36.658 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-1963-g61996d73-1focal 2023-01-25T06:27:36.659 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1963-g61996d73-1focal 2023-01-25T06:27:36.660 INFO:teuthology.task.install:The correct ceph version 18.0.0-1963-g61996d73-1focal is installed. 2023-01-25T06:27:36.661 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-25T06:27:36.662 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:27:36.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T06:27:36.716 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-25T06:27:36.717 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:27:36.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T06:27:36.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T06:27:36.792 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-25T06:27:36.793 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:27:36.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T06:27:36.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T06:27:36.821 INFO:teuthology.run_tasks:Running task workunit... 2023-01-25T06:27:36.835 INFO:tasks.workunit:Pulling workunits from ref 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T06:27:36.836 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-25T06:27:36.836 INFO:tasks.workunit:timeout=3h 2023-01-25T06:27:36.836 INFO:tasks.workunit:cleanup=True 2023-01-25T06:27:36.837 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-25T06:27:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T06:27:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-25T06:27:36.871 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T06:27:36.877 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-25T06:27:36.878 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-25T06:27:36.884 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T06:27:36.892 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-25T06:28:17.857 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to '61996d73ab738798e4ee0055c86457f5c459da4a'. 2023-01-25T06:28:17.857 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.857 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-25T06:28:17.858 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-25T06:28:17.858 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2023-01-25T06:28:17.858 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.858 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-25T06:28:17.859 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-25T06:28:17.859 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.859 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2023-01-25T06:28:17.859 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.859 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2023-01-25T06:28:17.860 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.860 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2023-01-25T06:28:17.860 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.860 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-25T06:28:17.860 INFO:tasks.workunit.client.0.smithi040.stderr: 2023-01-25T06:28:17.861 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at 61996d73ab7 Merge branch 'wip-tserong-qa-crash-chown' of https://github.com/SUSE/ceph into wip-yuri2-testing-2023-01-23-0928 2023-01-25T06:28:17.862 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-25T06:28:17.913 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-01-25T06:28:17.913 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-25T06:28:17.918 INFO:tasks.workunit:Running workunits matching mgr on client.0... 2023-01-25T06:28:17.919 INFO:tasks.workunit:Running workunit mgr/balancer.sh... 2023-01-25T06:28:17.919 DEBUG:teuthology.orchestra.run.smithi040:workunit test mgr/balancer.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61996d73ab738798e4ee0055c86457f5c459da4a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh 2023-01-25T06:28:17.941 INFO:tasks.workunit.client.0.smithi040.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-01-25T06:28:17.957 INFO:tasks.workunit.client.0.smithi040.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-01-25T06:28:17.958 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T06:28:17.958 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-01-25T06:28:17.958 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: export PYTHONWARNINGS=ignore 2023-01-25T06:28:17.958 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: main: PYTHONWARNINGS=ignore 2023-01-25T06:28:17.959 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export CEPH_CONF=/dev/null 2023-01-25T06:28:17.959 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: CEPH_CONF=/dev/null 2023-01-25T06:28:17.959 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: unset CEPH_ARGS 2023-01-25T06:28:17.959 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: local code 2023-01-25T06:28:17.959 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: main: run td/balancer 2023-01-25T06:28:17.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:20: run: local dir=td/balancer 2023-01-25T06:28:17.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:21: run: shift 2023-01-25T06:28:17.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:23: run: export CEPH_MON=127.0.0.1:7102 2023-01-25T06:28:17.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:23: run: CEPH_MON=127.0.0.1:7102 2023-01-25T06:28:17.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:24: run: export CEPH_ARGS 2023-01-25T06:28:17.961 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:25: run: uuidgen 2023-01-25T06:28:17.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:25: run: CEPH_ARGS+='--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none ' 2023-01-25T06:28:17.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7102 ' 2023-01-25T06:28:17.963 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:28: run: set 2023-01-25T06:28:17.963 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-01-25T06:28:17.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:28: run: local 'funcs=TEST_balancer 2023-01-25T06:28:17.964 INFO:tasks.workunit.client.0.smithi040.stderr:TEST_balancer2' 2023-01-25T06:28:17.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:29: run: for func in $funcs 2023-01-25T06:28:17.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:30: run: TEST_balancer td/balancer 2023-01-25T06:28:17.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:38: TEST_balancer: local dir=td/balancer 2023-01-25T06:28:17.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:40: TEST_balancer: setup td/balancer 2023-01-25T06:28:17.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/balancer 2023-01-25T06:28:17.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/balancer 2023-01-25T06:28:17.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/balancer 2023-01-25T06:28:17.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T06:28:17.966 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/balancer KILL 2023-01-25T06:28:17.966 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T06:28:17.967 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T06:28:17.967 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T06:28:17.967 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T06:28:17.967 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T06:28:17.967 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T06:28:17.968 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T06:28:17.968 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T06:28:17.969 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T06:28:17.970 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T06:28:17.971 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T06:28:17.971 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T06:28:17.972 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:28:17.972 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T06:28:17.973 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T06:28:17.973 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:28:17.974 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:28:17.975 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T06:28:17.975 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/balancer 2023-01-25T06:28:17.977 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T06:28:17.977 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:17.978 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:17.978 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22991 2023-01-25T06:28:17.979 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T06:28:17.979 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T06:28:17.979 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/balancer 2023-01-25T06:28:17.981 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T06:28:17.981 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:17.981 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:17.982 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.22991 2023-01-25T06:28:17.983 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T06:28:17.984 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-01-25T06:28:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-01-25T06:28:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T06:28:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/balancer 1' TERM HUP INT 2023-01-25T06:28:17.985 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:41: TEST_balancer: run_mon td/balancer a 2023-01-25T06:28:17.986 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/balancer 2023-01-25T06:28:17.986 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T06:28:17.986 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T06:28:17.986 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T06:28:17.986 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/balancer/a 2023-01-25T06:28:17.987 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/balancer/a --run-dir=td/balancer 2023-01-25T06:28:18.500 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T06:28:18.500 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:28:18.500 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:28:18.501 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:28:18.501 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:18.502 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:18.502 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:18.502 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/balancer/a '--log-file=td/balancer/$name.log' '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --mon-cluster-log-file=td/balancer/log --run-dir=td/balancer '--pid-file=td/balancer/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-01-25T06:28:18.789 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T06:28:18.791 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T06:28:18.791 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T06:28:18.791 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T06:28:18.791 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T06:28:18.792 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T06:28:18.793 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T06:28:18.793 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T06:28:18.793 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T06:28:18.793 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T06:28:18.793 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:18.794 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:18.794 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.22991/ceph-mon.a.asok 2023-01-25T06:28:18.794 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T06:28:18.795 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.22991/ceph-mon.a.asok config get fsid 2023-01-25T06:28:18.899 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T06:28:18.900 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T06:28:18.900 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T06:28:18.900 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T06:28:18.901 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T06:28:18.902 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T06:28:18.902 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T06:28:18.902 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T06:28:18.902 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T06:28:18.902 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:18.903 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:18.903 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.22991/ceph-mon.a.asok 2023-01-25T06:28:18.903 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T06:28:18.904 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.22991/ceph-mon.a.asok config get mon_host 2023-01-25T06:28:19.007 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:42: TEST_balancer: run_mgr td/balancer x 2023-01-25T06:28:19.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/balancer 2023-01-25T06:28:19.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T06:28:19.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T06:28:19.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T06:28:19.009 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/balancer/x 2023-01-25T06:28:19.009 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T06:28:19.912 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T06:28:19.912 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:28:19.912 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:28:19.913 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:28:19.913 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:19.913 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:19.913 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:19.914 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T06:28:19.921 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/balancer/x '--log-file=td/balancer/$name.log' '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --run-dir=td/balancer '--pid-file=td/balancer/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T06:28:20.015 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:43: TEST_balancer: run_osd td/balancer 0 2023-01-25T06:28:20.015 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:28:20.016 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:28:20.016 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T06:28:20.017 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:28:20.017 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/0 2023-01-25T06:28:20.018 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 ' 2023-01-25T06:28:20.018 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:28:20.019 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:28:20.019 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:28:20.019 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/0' 2023-01-25T06:28:20.020 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/0/journal' 2023-01-25T06:28:20.020 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:28:20.020 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:28:20.021 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:28:20.021 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:28:20.021 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:28:20.021 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:28:20.022 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:28:20.022 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:20.023 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:20.023 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:20.023 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:20.024 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:28:20.024 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:28:20.024 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:28:20.024 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:28:20.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:28:20.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:28:20.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:28:20.026 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:28:20.026 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:28:20.026 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:28:20.026 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/0 2023-01-25T06:28:20.027 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:28:20.027 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=75d93e2a-1c82-4426-92e3-df6e5253bd4c 2023-01-25T06:28:20.028 INFO:tasks.workunit.client.0.smithi040.stdout:add osd0 75d93e2a-1c82-4426-92e3-df6e5253bd4c 2023-01-25T06:28:20.028 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 75d93e2a-1c82-4426-92e3-df6e5253bd4c' 2023-01-25T06:28:20.029 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:28:20.039 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAEzNBjtA0wAhAAQlPApI3Udy7axxUBU9lWow== 2023-01-25T06:28:20.040 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAEzNBjtA0wAhAAQlPApI3Udy7axxUBU9lWow=="}' 2023-01-25T06:28:20.040 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 75d93e2a-1c82-4426-92e3-df6e5253bd4c -i td/balancer/0/new.json 2023-01-25T06:28:20.417 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:28:20.431 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/0/new.json 2023-01-25T06:28:20.433 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/0 --osd-journal=td/balancer/0/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAEzNBjtA0wAhAAQlPApI3Udy7axxUBU9lWow== --osd-uuid 75d93e2a-1c82-4426-92e3-df6e5253bd4c 2023-01-25T06:28:20.471 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.466+0000 7f4ccab93680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:20.499 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.494+0000 7f4ccab93680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:20.501 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.498+0000 7f4ccab93680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:20.502 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.498+0000 7f4ccab93680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:28:20.502 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.498+0000 7f4ccab93680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:28:20.503 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.498+0000 7f4ccab93680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:28:20.624 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:20.622+0000 7f4ccab93680 -1 bluestore(td/balancer/0) _read_fsid unparsable uuid 2023-01-25T06:28:26.830 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/0/keyring 2023-01-25T06:28:26.830 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:28:26.831 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T06:28:26.832 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd0 key to auth repository 2023-01-25T06:28:26.832 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:28:27.058 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T06:28:27.059 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.0 2023-01-25T06:28:27.060 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/0 --osd-journal=td/balancer/0/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:28:27.060 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:28:27.060 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:28:27.060 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:28:27.090 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:27.086+0000 7fe8a9630680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:27.094 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:27.090+0000 7fe8a9630680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:27.097 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:27.094+0000 7fe8a9630680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:27.293 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T06:28:27.294 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:28:27.294 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T06:28:27.295 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:28:27.295 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:28:27.295 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:27.296 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:28:27.296 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:28:27.297 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:27.298 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:27.515 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:27.655 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:27.650+0000 7fe8a9630680 -1 Falling back to public interface 2023-01-25T06:28:28.517 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:28.517 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:28.518 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:28:28.518 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:28:28.518 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:28.519 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:29.033 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:30.006 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:30.002+0000 7fe8a9630680 -1 osd.0 0 log_to_monitors true 2023-01-25T06:28:30.035 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:30.036 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:30.037 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:28:30.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:28:30.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:30.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:30.508 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:31.509 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:31.510 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:31.510 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:28:31.511 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:28:31.511 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:31.511 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:32.022 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:33.023 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:33.024 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:33.024 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:28:33.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:28:33.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:33.025 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:33.443 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:34.445 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:34.445 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:34.445 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:28:34.446 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:28:34.446 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:34.446 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:34.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:35.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:35.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:35.964 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:28:35.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:28:35.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:35.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:36.370 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:37.371 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:37.372 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:37.372 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:28:37.373 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:28:37.373 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:37.373 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:37.798 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:38.800 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:38.800 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:38.800 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:28:38.801 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:28:38.801 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:38.802 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:39.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:40.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:40.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:40.267 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:28:40.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:28:40.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:40.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:40.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:41.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:41.787 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:41.787 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:28:41.788 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:28:41.788 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:41.788 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:42.633 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:43.634 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:43.635 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:43.635 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:28:43.636 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:28:43.636 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:43.636 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:44.597 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:45.599 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:45.599 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:45.600 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:28:45.600 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:28:45.601 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:45.601 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:46.321 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:47.323 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:47.323 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:47.323 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:28:47.324 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:28:47.325 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:47.325 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:47.917 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:48.918 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:48.919 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:48.919 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:28:48.920 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:28:48.920 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:48.920 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:49.784 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:50.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:50.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:50.787 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:28:50.787 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:28:50.788 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:50.788 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:52.219 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:53.221 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:53.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:53.222 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:28:53.223 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:28:53.223 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:53.223 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:53.691 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:28:54.692 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:28:54.692 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:28:54.693 INFO:tasks.workunit.client.0.smithi040.stdout:17 2023-01-25T06:28:54.693 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T06:28:54.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:28:54.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:28:55.173 INFO:tasks.workunit.client.0.smithi040.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/24026,v1:127.0.0.1:6801/24026] [v2:127.0.0.1:6802/24026,v1:127.0.0.1:6803/24026] exists,up 75d93e2a-1c82-4426-92e3-df6e5253bd4c 2023-01-25T06:28:55.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:28:55.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:28:55.176 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:28:55.176 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:44: TEST_balancer: run_osd td/balancer 1 2023-01-25T06:28:55.176 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:28:55.176 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:28:55.176 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T06:28:55.177 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:28:55.177 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/1 2023-01-25T06:28:55.177 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 ' 2023-01-25T06:28:55.177 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:28:55.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:28:55.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:28:55.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/1' 2023-01-25T06:28:55.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/1/journal' 2023-01-25T06:28:55.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:28:55.179 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:28:55.180 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:28:55.180 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:28:55.180 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:55.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:28:55.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:28:55.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:28:55.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:28:55.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:28:55.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:28:55.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:28:55.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:28:55.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:28:55.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:28:55.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:28:55.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/1 2023-01-25T06:28:55.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:28:55.184 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1b6d8504-6628-4118-bbb3-8290c7601869 2023-01-25T06:28:55.184 INFO:tasks.workunit.client.0.smithi040.stdout:add osd1 1b6d8504-6628-4118-bbb3-8290c7601869 2023-01-25T06:28:55.185 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 1b6d8504-6628-4118-bbb3-8290c7601869' 2023-01-25T06:28:55.185 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:28:55.203 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAnzNBjzkcEDBAA3AzHhze6wruIP+TLgqIZYA== 2023-01-25T06:28:55.203 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAnzNBjzkcEDBAA3AzHhze6wruIP+TLgqIZYA=="}' 2023-01-25T06:28:55.203 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1b6d8504-6628-4118-bbb3-8290c7601869 -i td/balancer/1/new.json 2023-01-25T06:28:55.769 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:28:55.784 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/1/new.json 2023-01-25T06:28:55.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/1 --osd-journal=td/balancer/1/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAnzNBjzkcEDBAA3AzHhze6wruIP+TLgqIZYA== --osd-uuid 1b6d8504-6628-4118-bbb3-8290c7601869 2023-01-25T06:28:55.820 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.819+0000 7f7833650680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:55.824 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.823+0000 7f7833650680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:55.825 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.823+0000 7f7833650680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:28:55.827 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.823+0000 7f7833650680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:28:55.827 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.823+0000 7f7833650680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:28:55.827 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.823+0000 7f7833650680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:28:55.927 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:28:55.923+0000 7f7833650680 -1 bluestore(td/balancer/1) _read_fsid unparsable uuid 2023-01-25T06:29:02.217 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/1/keyring 2023-01-25T06:29:02.217 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:29:02.219 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T06:29:02.219 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd1 key to auth repository 2023-01-25T06:29:02.220 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:29:02.784 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.1 2023-01-25T06:29:02.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T06:29:02.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/1 --osd-journal=td/balancer/1/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:29:02.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:29:02.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:29:02.786 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:29:02.818 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:02.815+0000 7fd008997680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:02.823 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:02.823+0000 7fd008997680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:02.825 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:02.823+0000 7fd008997680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:03.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T06:29:03.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:29:03.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T06:29:03.223 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:29:03.223 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:29:03.224 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:29:03.224 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:03.224 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:29:03.224 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:03.225 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:03.401 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:03.399+0000 7fd008997680 -1 Falling back to public interface 2023-01-25T06:29:03.631 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:04.632 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:04.633 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:04.633 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:29:04.634 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:29:04.634 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:04.634 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:05.045 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:05.579 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:05.579+0000 7fd008997680 -1 osd.1 0 log_to_monitors true 2023-01-25T06:29:06.047 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:06.048 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:06.048 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:29:06.049 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:29:06.049 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:06.049 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:06.692 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:07.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:07.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:07.695 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:29:07.695 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:29:07.696 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:07.696 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:08.131 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:09.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:09.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:09.133 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:29:09.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:29:09.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:09.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:09.554 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:10.556 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:10.557 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:10.557 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:29:10.558 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:29:10.558 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:10.558 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:10.999 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:12.001 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:12.001 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:12.002 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:29:12.003 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:29:12.003 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:12.003 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:12.466 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:13.468 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:13.468 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:13.468 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:29:13.469 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:29:13.469 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:13.470 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:13.904 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:14.906 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:14.906 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:14.906 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:29:14.907 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:29:14.907 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:14.908 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:15.455 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:16.456 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:16.457 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:16.457 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:29:16.458 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:29:16.458 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:16.458 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:16.909 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:17.911 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:17.912 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:17.912 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:29:17.913 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:29:17.913 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:17.913 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:18.678 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:19.680 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:19.681 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:19.681 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:29:19.682 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:29:19.683 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:19.683 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:20.622 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:21.623 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:21.624 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:21.624 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:29:21.625 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:29:21.626 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:21.627 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:22.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:23.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:23.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:23.184 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:29:23.185 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:29:23.185 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:23.185 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:23.740 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:24.742 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:24.742 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:24.744 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:29:24.744 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:29:24.745 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:24.745 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:25.189 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:26.191 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:26.192 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:26.192 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:29:26.193 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:29:26.193 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:26.193 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:27.056 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:28.057 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:28.057 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:28.058 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:29:28.058 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:29:28.059 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:28.059 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:28.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:29.650 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:29.651 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:29.651 INFO:tasks.workunit.client.0.smithi040.stdout:17 2023-01-25T06:29:29.652 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 17 2023-01-25T06:29:29.652 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:29.652 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:29:30.129 INFO:tasks.workunit.client.0.smithi040.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25917,v1:127.0.0.1:6811/25917] [v2:127.0.0.1:6812/25917,v1:127.0.0.1:6813/25917] exists,up 1b6d8504-6628-4118-bbb3-8290c7601869 2023-01-25T06:29:30.131 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:29:30.131 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:29:30.132 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:29:30.132 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:45: TEST_balancer: run_osd td/balancer 2 2023-01-25T06:29:30.132 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:29:30.132 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:29:30.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T06:29:30.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:29:30.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/2 2023-01-25T06:29:30.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 ' 2023-01-25T06:29:30.133 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:29:30.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:29:30.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:29:30.134 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/2' 2023-01-25T06:29:30.135 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/2/journal' 2023-01-25T06:29:30.135 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:29:30.135 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:29:30.135 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:29:30.135 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:29:30.136 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:29:30.136 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:29:30.137 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:29:30.138 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:29:30.138 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:29:30.138 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:29:30.138 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:29:30.139 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:29:30.139 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:29:30.139 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:29:30.139 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:29:30.139 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:29:30.140 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:29:30.140 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:29:30.140 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:29:30.140 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:29:30.140 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:29:30.141 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/2 2023-01-25T06:29:30.141 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:29:30.141 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=b29bde41-41ba-4443-be14-bfab28612aee 2023-01-25T06:29:30.142 INFO:tasks.workunit.client.0.smithi040.stdout:add osd2 b29bde41-41ba-4443-be14-bfab28612aee 2023-01-25T06:29:30.142 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 b29bde41-41ba-4443-be14-bfab28612aee' 2023-01-25T06:29:30.143 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:29:30.164 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBKzNBjHnu8CRAAGAQO0CSJtUJVW8T3mNB3Cw== 2023-01-25T06:29:30.164 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBKzNBjHnu8CRAAGAQO0CSJtUJVW8T3mNB3Cw=="}' 2023-01-25T06:29:30.164 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new b29bde41-41ba-4443-be14-bfab28612aee -i td/balancer/2/new.json 2023-01-25T06:29:30.667 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:29:30.680 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/2/new.json 2023-01-25T06:29:30.681 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/2 --osd-journal=td/balancer/2/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBKzNBjHnu8CRAAGAQO0CSJtUJVW8T3mNB3Cw== --osd-uuid b29bde41-41ba-4443-be14-bfab28612aee 2023-01-25T06:29:30.714 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.711+0000 7f94111ac680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:30.744 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.743+0000 7f94111ac680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:30.747 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.747+0000 7f94111ac680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:30.748 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.747+0000 7f94111ac680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:29:30.748 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.747+0000 7f94111ac680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:29:30.748 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.747+0000 7f94111ac680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:29:30.847 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:30.847+0000 7f94111ac680 -1 bluestore(td/balancer/2) _read_fsid unparsable uuid 2023-01-25T06:29:36.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/2/keyring 2023-01-25T06:29:36.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:29:36.777 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T06:29:36.778 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd2 key to auth repository 2023-01-25T06:29:36.778 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:29:37.354 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T06:29:37.355 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.2 2023-01-25T06:29:37.356 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:29:37.356 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/2 --osd-journal=td/balancer/2/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:29:37.356 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:29:37.356 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:29:37.387 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:37.382+0000 7fbfef0dc680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:37.407 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:37.406+0000 7fbfef0dc680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:37.408 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:37.406+0000 7fbfef0dc680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:29:37.846 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T06:29:37.846 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:29:37.846 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T06:29:37.847 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:29:37.847 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:29:37.847 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:37.847 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:29:37.848 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:29:37.848 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:37.848 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:37.964 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:37.962+0000 7fbfef0dc680 -1 Falling back to public interface 2023-01-25T06:29:38.244 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:39.246 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:39.246 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:39.246 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:29:39.247 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:29:39.247 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:39.247 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:39.662 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:40.447 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:29:40.446+0000 7fbfef0dc680 -1 osd.2 0 log_to_monitors true 2023-01-25T06:29:40.664 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:40.664 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:40.665 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:29:40.665 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:29:40.665 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:40.666 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:41.153 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:42.155 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:42.155 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:42.156 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:29:42.156 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:29:42.156 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:42.157 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:42.593 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:43.595 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:43.596 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:43.596 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:29:43.597 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:29:43.597 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:43.598 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:44.027 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:45.029 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:45.029 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:45.030 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:29:45.030 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:29:45.031 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:45.031 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:45.793 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:46.795 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:46.796 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:46.796 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:29:46.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:29:46.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:46.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:47.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:48.264 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:48.264 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:48.265 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:29:48.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:29:48.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:48.269 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:48.682 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:49.684 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:49.685 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:49.685 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:29:49.685 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:29:49.686 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:49.686 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:50.091 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:51.093 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:51.094 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:51.094 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:29:51.095 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:29:51.095 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:51.095 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:51.506 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:52.508 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:52.508 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:52.509 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:29:52.509 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:29:52.509 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:52.510 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:53.114 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:54.116 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:54.117 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:54.117 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:29:54.117 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:29:54.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:54.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:54.831 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:55.833 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:55.833 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:55.834 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:29:55.834 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:29:55.835 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:55.835 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:56.781 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:57.783 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:57.784 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:57.784 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:29:57.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:29:57.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:57.785 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:29:58.600 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:29:59.602 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:29:59.603 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:29:59.603 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:29:59.604 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:29:59.604 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:29:59.604 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:30:00.427 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:01.429 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:01.430 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:01.431 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:30:01.432 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:30:01.432 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:01.432 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:30:02.248 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:03.250 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:03.250 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:03.250 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:30:03.251 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:30:03.251 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:03.251 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:30:03.774 INFO:tasks.workunit.client.0.smithi040.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/27677,v1:127.0.0.1:6819/27677] [v2:127.0.0.1:6820/27677,v1:127.0.0.1:6821/27677] exists,up b29bde41-41ba-4443-be14-bfab28612aee 2023-01-25T06:30:03.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:30:03.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:30:03.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:30:03.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:46: TEST_balancer: create_pool test1 8 2023-01-25T06:30:03.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 8 2023-01-25T06:30:04.469 INFO:tasks.workunit.client.0.smithi040.stderr:pool 'test1' already exists 2023-01-25T06:30:04.483 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T06:30:05.485 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:47: TEST_balancer: create_pool test2 8 2023-01-25T06:30:05.485 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 8 2023-01-25T06:30:06.479 INFO:tasks.workunit.client.0.smithi040.stderr:pool 'test2' already exists 2023-01-25T06:30:06.488 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T06:30:07.490 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:49: TEST_balancer: wait_for_clean 2023-01-25T06:30:07.490 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T06:30:07.491 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T06:30:07.491 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T06:30:07.491 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T06:30:07.491 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T06:30:07.492 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T06:30:07.493 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T06:30:07.493 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T06:30:07.493 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T06:30:07.493 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T06:30:07.696 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T06:30:07.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T06:30:07.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T06:30:07.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T06:30:07.698 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T06:30:08.142 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T06:30:08.143 INFO:tasks.workunit.client.0.smithi040.stderr:1 2023-01-25T06:30:08.143 INFO:tasks.workunit.client.0.smithi040.stderr:2' 2023-01-25T06:30:08.143 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T06:30:08.143 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:30:08.143 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T06:30:08.290 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836503 2023-01-25T06:30:08.291 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836503 2023-01-25T06:30:08.291 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836503' 2023-01-25T06:30:08.291 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:30:08.292 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T06:30:08.550 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672980 2023-01-25T06:30:08.551 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672980 2023-01-25T06:30:08.551 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2023-01-25T06:30:08.551 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:30:08.552 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T06:30:08.712 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509452 2023-01-25T06:30:08.712 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509452 2023-01-25T06:30:08.713 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509452' 2023-01-25T06:30:08.713 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:30:08.713 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836503 2023-01-25T06:30:08.714 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:30:08.715 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T06:30:08.717 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836503 2023-01-25T06:30:08.717 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:30:08.718 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836503 2023-01-25T06:30:08.718 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2023-01-25T06:30:08.719 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.0 seq 21474836503 2023-01-25T06:30:08.719 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:30:09.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836501 -lt 21474836503 2023-01-25T06:30:09.178 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T06:30:10.180 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T06:30:10.181 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:30:10.604 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836503 -lt 21474836503 2023-01-25T06:30:10.604 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:30:10.605 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672980 2023-01-25T06:30:10.605 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:30:10.607 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T06:30:10.609 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672980 2023-01-25T06:30:10.609 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:30:10.610 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672980 2023-01-25T06:30:10.610 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2023-01-25T06:30:10.610 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.1 seq 42949672980 2023-01-25T06:30:10.611 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T06:30:11.062 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672981 -lt 42949672980 2023-01-25T06:30:11.063 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:30:11.064 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509452 2023-01-25T06:30:11.064 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:30:11.066 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T06:30:11.068 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509452 2023-01-25T06:30:11.068 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:30:11.070 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509452 2023-01-25T06:30:11.071 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.2 seq 64424509452 2023-01-25T06:30:11.071 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2023-01-25T06:30:11.072 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T06:30:11.494 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509453 -lt 64424509452 2023-01-25T06:30:11.496 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T06:30:11.496 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:30:11.497 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:30:11.999 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 16 == 0 2023-01-25T06:30:12.000 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T06:30:12.001 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T06:30:12.002 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T06:30:12.002 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T06:30:12.002 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T06:30:12.003 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T06:30:12.003 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T06:30:12.448 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=16 2023-01-25T06:30:12.449 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T06:30:12.449 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:30:12.450 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:30:12.983 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 16 = 16 2023-01-25T06:30:12.983 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T06:30:12.983 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T06:30:12.983 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:51: TEST_balancer: ceph pg dump pgs 2023-01-25T06:30:13.397 INFO:tasks.workunit.client.0.smithi040.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-01-25T06:30:13.398 INFO:tasks.workunit.client.0.smithi040.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.110300+0000 0'0 25:18 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T10:02:00.287123+0000 0 0 2023-01-25T06:30:13.398 INFO:tasks.workunit.client.0.smithi040.stdout:2.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.328749+0000 0'0 25:13 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T07:48:45.912008+0000 0 0 2023-01-25T06:30:13.398 INFO:tasks.workunit.client.0.smithi040.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.488629+0000 0'0 26:17 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T17:08:07.392332+0000 0 0 2023-01-25T06:30:13.399 INFO:tasks.workunit.client.0.smithi040.stdout:2.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:07.377439+0000 0'0 25:12 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T07:42:36.305443+0000 0 0 2023-01-25T06:30:13.399 INFO:tasks.workunit.client.0.smithi040.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.487340+0000 0'0 25:17 [0,1,2] 0 [0,1,2] 0 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T10:31:33.522688+0000 0 0 2023-01-25T06:30:13.399 INFO:tasks.workunit.client.0.smithi040.stdout:2.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.204641+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T12:26:48.533478+0000 0 0 2023-01-25T06:30:13.399 INFO:tasks.workunit.client.0.smithi040.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.220762+0000 0'0 25:18 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T13:37:56.730753+0000 0 0 2023-01-25T06:30:13.400 INFO:tasks.workunit.client.0.smithi040.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.204688+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T11:15:42.508818+0000 0 0 2023-01-25T06:30:13.400 INFO:tasks.workunit.client.0.smithi040.stdout:2.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.329283+0000 0'0 25:13 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T11:08:30.114985+0000 0 0 2023-01-25T06:30:13.400 INFO:tasks.workunit.client.0.smithi040.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.221291+0000 0'0 25:18 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T18:29:24.557049+0000 0 0 2023-01-25T06:30:13.400 INFO:tasks.workunit.client.0.smithi040.stdout:2.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.328600+0000 0'0 25:13 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T13:51:59.883829+0000 0 0 2023-01-25T06:30:13.401 INFO:tasks.workunit.client.0.smithi040.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.488515+0000 0'0 26:17 [2,0,1] 2 [2,0,1] 2 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T10:55:38.029740+0000 0 0 2023-01-25T06:30:13.401 INFO:tasks.workunit.client.0.smithi040.stdout:2.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.329765+0000 0'0 25:13 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T13:00:21.619784+0000 0 0 2023-01-25T06:30:13.401 INFO:tasks.workunit.client.0.smithi040.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.224116+0000 0'0 25:18 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T12:18:34.739280+0000 0 0 2023-01-25T06:30:13.401 INFO:tasks.workunit.client.0.smithi040.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:05.224025+0000 0'0 25:18 [1,2,0] 1 [1,2,0] 1 0'0 2023-01-25T06:30:04.130542+0000 0'0 2023-01-25T06:30:04.130542+0000 0 0 periodic scrub scheduled @ 2023-01-26T14:23:20.775082+0000 0 0 2023-01-25T06:30:13.402 INFO:tasks.workunit.client.0.smithi040.stdout:2.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-25T06:30:08.330200+0000 0'0 25:13 [1,0,2] 1 [1,0,2] 1 0'0 2023-01-25T06:30:06.164764+0000 0'0 2023-01-25T06:30:06.164764+0000 0 0 periodic scrub scheduled @ 2023-01-26T08:12:25.626879+0000 0 0 2023-01-25T06:30:13.402 INFO:tasks.workunit.client.0.smithi040.stdout: 2023-01-25T06:30:13.402 INFO:tasks.workunit.client.0.smithi040.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-01-25T06:30:13.403 INFO:tasks.workunit.client.0.smithi040.stderr:dumped pgs 2023-01-25T06:30:13.409 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:52: TEST_balancer: ceph balancer status 2023-01-25T06:30:13.787 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:13.787 INFO:tasks.workunit.client.0.smithi040.stdout: "active": true, 2023-01-25T06:30:13.787 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.000364", 2023-01-25T06:30:13.787 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:29:28 2023", 2023-01-25T06:30:13.788 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "upmap", 2023-01-25T06:30:13.788 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": false, 2023-01-25T06:30:13.788 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "No pools available", 2023-01-25T06:30:13.788 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:13.788 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:13.799 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:53: TEST_balancer: ceph balancer status 2023-01-25T06:30:13.800 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:53: TEST_balancer: jq .mode 2023-01-25T06:30:14.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:53: TEST_balancer: eval 'MODE="upmap"' 2023-01-25T06:30:14.182 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:53: TEST_balancer: MODE=upmap 2023-01-25T06:30:14.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:54: TEST_balancer: test upmap = upmap 2023-01-25T06:30:14.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:55: TEST_balancer: ceph balancer status 2023-01-25T06:30:14.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:55: TEST_balancer: jq .active 2023-01-25T06:30:14.583 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:55: TEST_balancer: ACTIVE=true 2023-01-25T06:30:14.583 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:56: TEST_balancer: test true = true 2023-01-25T06:30:14.584 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:58: TEST_balancer: ceph balancer ls 2023-01-25T06:30:14.960 INFO:tasks.workunit.client.0.smithi040.stdout:[] 2023-01-25T06:30:14.977 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:59: TEST_balancer: ceph balancer ls 2023-01-25T06:30:15.374 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:59: TEST_balancer: PLANS='[]' 2023-01-25T06:30:15.375 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:60: TEST_balancer: test '[]' = '[]' 2023-01-25T06:30:15.375 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:61: TEST_balancer: ceph balancer eval 2023-01-25T06:30:15.768 INFO:tasks.workunit.client.0.smithi040.stdout:current cluster score 0.000000 (lower is better) 2023-01-25T06:30:15.783 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:62: TEST_balancer: ceph balancer eval 2023-01-25T06:30:16.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:62: TEST_balancer: EVAL='current cluster score 0.000000 (lower is better)' 2023-01-25T06:30:16.184 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:63: TEST_balancer: test 'current cluster score 0.000000 (lower is better)' = 'current cluster score 0.000000 (lower is better)' 2023-01-25T06:30:16.184 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:64: TEST_balancer: ceph balancer eval-verbose 2023-01-25T06:30:16.589 INFO:tasks.workunit.client.0.smithi040.stdout:current cluster 2023-01-25T06:30:16.589 INFO:tasks.workunit.client.0.smithi040.stdout:target_by_root {'default': {0: 0.3333333333333333, 1: 0.3333333333333333, 2: 0.3333333333333333}} 2023-01-25T06:30:16.589 INFO:tasks.workunit.client.0.smithi040.stdout:actual_by_pool {'test1': {'pgs': {1: 0.3333333333333333, 0: 0.3333333333333333, 2: 0.3333333333333333}, 'objects': {1: 0.0, 0: 0.0, 2: 0.0}, 'bytes': {1: 0.0, 0: 0.0, 2: 0.0}}, 'test2': {'pgs': {2: 0.3333333333333333, 1: 0.3333333333333333, 0: 0.3333333333333333}, 'objects': {2: 0.0, 1: 0.0, 0: 0.0}, 'bytes': {2: 0.0, 1: 0.0, 0: 0.0}}} 2023-01-25T06:30:16.590 INFO:tasks.workunit.client.0.smithi040.stdout:actual_by_root {'default': {'pgs': {0: 0.3333333333333333, 1: 0.3333333333333333, 2: 0.3333333333333333}, 'objects': {0: 0.0, 1: 0.0, 2: 0.0}, 'bytes': {0: 0.0, 1: 0.0, 2: 0.0}}} 2023-01-25T06:30:16.590 INFO:tasks.workunit.client.0.smithi040.stdout:count_by_pool {'test1': {'pgs': {1: 8, 0: 8, 2: 8}, 'objects': {1: 0, 0: 0, 2: 0}, 'bytes': {1: 0, 0: 0, 2: 0}}, 'test2': {'pgs': {2: 8, 1: 8, 0: 8}, 'objects': {2: 0, 1: 0, 0: 0}, 'bytes': {2: 0, 1: 0, 0: 0}}} 2023-01-25T06:30:16.590 INFO:tasks.workunit.client.0.smithi040.stdout:count_by_root {'default': {'pgs': {0: 16.0, 1: 16.0, 2: 16.0}, 'objects': {0: 0.0, 1: 0.0, 2: 0.0}, 'bytes': {0: 0.0, 1: 0.0, 2: 0.0}}} 2023-01-25T06:30:16.590 INFO:tasks.workunit.client.0.smithi040.stdout:total_by_pool {'test1': {'pgs': 24, 'objects': 0, 'bytes': 0}, 'test2': {'pgs': 24, 'objects': 0, 'bytes': 0}} 2023-01-25T06:30:16.590 INFO:tasks.workunit.client.0.smithi040.stdout:total_by_root {'default': {'pgs': 48, 'objects': 0, 'bytes': 0}} 2023-01-25T06:30:16.591 INFO:tasks.workunit.client.0.smithi040.stdout:stats_by_root {'default': {'pgs': {'max': 16.0, 'min': 16.0, 'avg': 16.0, 'stddev': 0.0, 'sum_weight': 0.0, 'score': 0.0}, 'objects': {'max': 0, 'min': 0, 'avg': 0, 'stddev': 0, 'sum_weight': 0, 'score': 0}, 'bytes': {'max': 0, 'min': 0, 'avg': 0, 'stddev': 0, 'sum_weight': 0, 'score': 0}}} 2023-01-25T06:30:16.591 INFO:tasks.workunit.client.0.smithi040.stdout:score_by_pool {} 2023-01-25T06:30:16.591 INFO:tasks.workunit.client.0.smithi040.stdout:score_by_root {'default': {'pgs': 0.0, 'objects': 0, 'bytes': 0}} 2023-01-25T06:30:16.591 INFO:tasks.workunit.client.0.smithi040.stdout:score 0.000000 (lower is better) 2023-01-25T06:30:16.602 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:66: TEST_balancer: ceph balancer pool add test1 2023-01-25T06:30:17.104 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:67: TEST_balancer: ceph balancer pool add test2 2023-01-25T06:30:17.595 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:68: TEST_balancer: ceph balancer pool ls 2023-01-25T06:30:17.986 INFO:tasks.workunit.client.0.smithi040.stdout:[ 2023-01-25T06:30:17.986 INFO:tasks.workunit.client.0.smithi040.stdout: "test1", 2023-01-25T06:30:17.986 INFO:tasks.workunit.client.0.smithi040.stdout: "test2" 2023-01-25T06:30:17.986 INFO:tasks.workunit.client.0.smithi040.stdout:] 2023-01-25T06:30:17.999 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:69: TEST_balancer: ceph balancer pool ls 2023-01-25T06:30:17.999 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:69: TEST_balancer: jq 'sort | .[0]' 2023-01-25T06:30:18.425 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:69: TEST_balancer: eval 'POOL="test1"' 2023-01-25T06:30:18.426 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:69: TEST_balancer: POOL=test1 2023-01-25T06:30:18.426 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:70: TEST_balancer: test test1 = test1 2023-01-25T06:30:18.427 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:71: TEST_balancer: ceph balancer pool ls 2023-01-25T06:30:18.427 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:71: TEST_balancer: jq 'sort | .[1]' 2023-01-25T06:30:18.838 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:71: TEST_balancer: eval 'POOL="test2"' 2023-01-25T06:30:18.838 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:71: TEST_balancer: POOL=test2 2023-01-25T06:30:18.839 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:72: TEST_balancer: test test2 = test2 2023-01-25T06:30:18.839 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:73: TEST_balancer: ceph balancer pool rm test1 2023-01-25T06:30:19.378 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:74: TEST_balancer: ceph balancer pool rm test2 2023-01-25T06:30:19.863 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:75: TEST_balancer: ceph balancer pool ls 2023-01-25T06:30:20.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:76: TEST_balancer: ceph balancer pool add test1 2023-01-25T06:30:20.764 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:78: TEST_balancer: ceph balancer mode crush-compat 2023-01-25T06:30:21.498 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:79: TEST_balancer: ceph balancer status 2023-01-25T06:30:21.908 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:21.908 INFO:tasks.workunit.client.0.smithi040.stdout: "active": true, 2023-01-25T06:30:21.909 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.000364", 2023-01-25T06:30:21.909 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:29:28 2023", 2023-01-25T06:30:21.909 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "crush-compat", 2023-01-25T06:30:21.909 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": false, 2023-01-25T06:30:21.910 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "No pools available", 2023-01-25T06:30:21.910 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:21.910 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:21.923 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:80: TEST_balancer: ceph balancer status 2023-01-25T06:30:21.923 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:80: TEST_balancer: jq .mode 2023-01-25T06:30:22.328 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:80: TEST_balancer: eval 'MODE="crush-compat"' 2023-01-25T06:30:22.329 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:80: TEST_balancer: MODE=crush-compat 2023-01-25T06:30:22.329 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:81: TEST_balancer: test crush-compat = crush-compat 2023-01-25T06:30:22.329 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:82: TEST_balancer: ceph balancer off 2023-01-25T06:30:22.803 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:83: TEST_balancer: ceph balancer optimize plan_crush test1 2023-01-25T06:30:23.101 INFO:tasks.workunit.client.0.smithi040.stderr:Error EALREADY: Distribution is already perfect 2023-01-25T06:30:23.105 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:84: TEST_balancer: ceph balancer status 2023-01-25T06:30:23.506 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:23.507 INFO:tasks.workunit.client.0.smithi040.stdout: "active": false, 2023-01-25T06:30:23.507 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.001171", 2023-01-25T06:30:23.507 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:30:23 2023", 2023-01-25T06:30:23.507 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "crush-compat", 2023-01-25T06:30:23.508 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": false, 2023-01-25T06:30:23.508 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "Distribution is already perfect", 2023-01-25T06:30:23.508 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:23.508 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:23.520 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:85: TEST_balancer: ceph balancer status 2023-01-25T06:30:23.521 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:85: TEST_balancer: jq .optimize_result 2023-01-25T06:30:23.922 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:85: TEST_balancer: eval 'RESULT="Distribution' is already 'perfect"' 2023-01-25T06:30:23.922 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:85: TEST_balancer: RESULT='Distribution is already perfect' 2023-01-25T06:30:23.922 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:86: TEST_balancer: test 'Distribution is already perfect' = 'Distribution is already perfect' 2023-01-25T06:30:23.923 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:88: TEST_balancer: ceph balancer on 2023-01-25T06:30:24.403 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:89: TEST_balancer: ceph balancer status 2023-01-25T06:30:24.404 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:89: TEST_balancer: jq .active 2023-01-25T06:30:24.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:89: TEST_balancer: ACTIVE=true 2023-01-25T06:30:24.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:90: TEST_balancer: test true = true 2023-01-25T06:30:24.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:91: TEST_balancer: sleep 2 2023-01-25T06:30:26.798 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:92: TEST_balancer: ceph balancer status 2023-01-25T06:30:27.191 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:27.191 INFO:tasks.workunit.client.0.smithi040.stdout: "active": true, 2023-01-25T06:30:27.192 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.001850", 2023-01-25T06:30:27.192 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:30:24 2023", 2023-01-25T06:30:27.192 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "crush-compat", 2023-01-25T06:30:27.192 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": false, 2023-01-25T06:30:27.192 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "Distribution is already perfect", 2023-01-25T06:30:27.193 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:27.193 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:27.203 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:93: TEST_balancer: ceph balancer off 2023-01-25T06:30:27.676 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:94: TEST_balancer: ceph balancer status 2023-01-25T06:30:27.677 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:94: TEST_balancer: jq .active 2023-01-25T06:30:28.074 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:94: TEST_balancer: ACTIVE=false 2023-01-25T06:30:28.074 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:95: TEST_balancer: test false = false 2023-01-25T06:30:28.074 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:96: TEST_balancer: sleep 2 2023-01-25T06:30:30.076 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:98: TEST_balancer: ceph balancer reset 2023-01-25T06:30:30.491 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:100: TEST_balancer: ceph balancer mode upmap 2023-01-25T06:30:30.970 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:101: TEST_balancer: ceph balancer status 2023-01-25T06:30:31.350 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:31.350 INFO:tasks.workunit.client.0.smithi040.stdout: "active": false, 2023-01-25T06:30:31.350 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.001850", 2023-01-25T06:30:31.350 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:30:24 2023", 2023-01-25T06:30:31.351 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "upmap", 2023-01-25T06:30:31.351 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": false, 2023-01-25T06:30:31.351 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "Distribution is already perfect", 2023-01-25T06:30:31.351 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:31.351 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:31.371 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:102: TEST_balancer: ceph balancer status 2023-01-25T06:30:31.371 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:102: TEST_balancer: jq .mode 2023-01-25T06:30:31.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:102: TEST_balancer: eval 'MODE="upmap"' 2023-01-25T06:30:31.777 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:102: TEST_balancer: MODE=upmap 2023-01-25T06:30:31.777 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:103: TEST_balancer: test upmap = upmap 2023-01-25T06:30:31.777 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:104: TEST_balancer: ceph balancer optimize plan_upmap 2023-01-25T06:30:32.065 INFO:tasks.workunit.client.0.smithi040.stderr:Error EALREADY: Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect 2023-01-25T06:30:32.068 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:105: TEST_balancer: ceph balancer status 2023-01-25T06:30:32.498 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:32.498 INFO:tasks.workunit.client.0.smithi040.stdout: "active": false, 2023-01-25T06:30:32.499 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.000940", 2023-01-25T06:30:32.499 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:30:32 2023", 2023-01-25T06:30:32.499 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "upmap", 2023-01-25T06:30:32.499 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": true, 2023-01-25T06:30:32.499 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2023-01-25T06:30:32.500 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:32.500 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:32.512 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:106: TEST_balancer: ceph balancer status 2023-01-25T06:30:32.513 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:106: TEST_balancer: jq .optimize_result 2023-01-25T06:30:32.906 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:106: TEST_balancer: eval 'RESULT="Unable' to find further optimization, or 'pool(s)' pg_num is decreasing, or distribution is already 'perfect"' 2023-01-25T06:30:32.907 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:106: TEST_balancer: RESULT='Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect' 2023-01-25T06:30:32.907 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:107: TEST_balancer: test 'Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect' = 'Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect' 2023-01-25T06:30:32.907 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:109: TEST_balancer: ceph balancer on 2023-01-25T06:30:33.399 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:110: TEST_balancer: ceph balancer status 2023-01-25T06:30:33.400 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:110: TEST_balancer: jq .active 2023-01-25T06:30:33.796 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:110: TEST_balancer: ACTIVE=true 2023-01-25T06:30:33.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:111: TEST_balancer: test true = true 2023-01-25T06:30:33.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:112: TEST_balancer: sleep 2 2023-01-25T06:30:35.797 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:113: TEST_balancer: ceph balancer status 2023-01-25T06:30:36.199 INFO:tasks.workunit.client.0.smithi040.stdout:{ 2023-01-25T06:30:36.199 INFO:tasks.workunit.client.0.smithi040.stdout: "active": true, 2023-01-25T06:30:36.199 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_duration": "0:00:00.000507", 2023-01-25T06:30:36.200 INFO:tasks.workunit.client.0.smithi040.stdout: "last_optimize_started": "Wed Jan 25 06:30:33 2023", 2023-01-25T06:30:36.200 INFO:tasks.workunit.client.0.smithi040.stdout: "mode": "upmap", 2023-01-25T06:30:36.200 INFO:tasks.workunit.client.0.smithi040.stdout: "no_optimization_needed": true, 2023-01-25T06:30:36.200 INFO:tasks.workunit.client.0.smithi040.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2023-01-25T06:30:36.200 INFO:tasks.workunit.client.0.smithi040.stdout: "plans": [] 2023-01-25T06:30:36.201 INFO:tasks.workunit.client.0.smithi040.stdout:} 2023-01-25T06:30:36.210 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:114: TEST_balancer: ceph balancer off 2023-01-25T06:30:36.710 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:115: TEST_balancer: ceph balancer status 2023-01-25T06:30:36.711 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:115: TEST_balancer: jq .active 2023-01-25T06:30:37.111 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:115: TEST_balancer: ACTIVE=false 2023-01-25T06:30:37.112 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:116: TEST_balancer: test false = false 2023-01-25T06:30:37.112 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:118: TEST_balancer: teardown td/balancer 2023-01-25T06:30:37.112 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/balancer 2023-01-25T06:30:37.113 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T06:30:37.113 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/balancer KILL 2023-01-25T06:30:37.114 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T06:30:37.114 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T06:30:37.114 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T06:30:37.114 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T06:30:37.114 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T06:30:37.227 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T06:30:37.227 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T06:30:37.228 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T06:30:37.229 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T06:30:37.231 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T06:30:37.231 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T06:30:37.232 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T06:30:37.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:30:37.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T06:30:37.234 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T06:30:37.235 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:30:37.236 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:30:37.238 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T06:30:37.238 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/balancer 2023-01-25T06:30:37.255 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T06:30:37.255 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:37.256 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:37.256 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22991 2023-01-25T06:30:37.257 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T06:30:37.257 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T06:30:37.258 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:29: run: for func in $funcs 2023-01-25T06:30:37.258 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:30: run: TEST_balancer2 td/balancer 2023-01-25T06:30:37.258 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:122: TEST_balancer2: local dir=td/balancer 2023-01-25T06:30:37.258 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:123: TEST_balancer2: TEST_PGS1=118 2023-01-25T06:30:37.258 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:124: TEST_balancer2: TEST_PGS2=132 2023-01-25T06:30:37.259 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:125: TEST_balancer2: expr 118 + 132 2023-01-25T06:30:37.259 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:125: TEST_balancer2: TOTAL_PGS=250 2023-01-25T06:30:37.259 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:126: TEST_balancer2: OSDS=5 2023-01-25T06:30:37.259 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:127: TEST_balancer2: DEFAULT_REPLICAS=3 2023-01-25T06:30:37.260 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:129: TEST_balancer2: expr '(' 118 '*' 3 ')' / 5 2023-01-25T06:30:37.260 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:129: TEST_balancer2: FINAL_PER_OSD1=70 2023-01-25T06:30:37.260 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:131: TEST_balancer2: expr '(' '(' 118 + 132 ')' '*' 3 ')' / 5 2023-01-25T06:30:37.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:131: TEST_balancer2: FINAL_PER_OSD2=150 2023-01-25T06:30:37.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:133: TEST_balancer2: CEPH_ARGS+='--osd_pool_default_pg_autoscale_mode=off ' 2023-01-25T06:30:37.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:134: TEST_balancer2: CEPH_ARGS+='--debug_osd=20 ' 2023-01-25T06:30:37.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:135: TEST_balancer2: setup td/balancer 2023-01-25T06:30:37.262 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/balancer 2023-01-25T06:30:37.263 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/balancer 2023-01-25T06:30:37.263 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/balancer 2023-01-25T06:30:37.263 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T06:30:37.263 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/balancer KILL 2023-01-25T06:30:37.265 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T06:30:37.265 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T06:30:37.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T06:30:37.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T06:30:37.266 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T06:30:37.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T06:30:37.267 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T06:30:37.269 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T06:30:37.269 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T06:30:37.270 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T06:30:37.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T06:30:37.271 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T06:30:37.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:30:37.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T06:30:37.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T06:30:37.274 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:30:37.275 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:30:37.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T06:30:37.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/balancer 2023-01-25T06:30:37.279 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T06:30:37.280 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:37.280 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:37.280 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22991 2023-01-25T06:30:37.281 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T06:30:37.282 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T06:30:37.282 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/balancer 2023-01-25T06:30:37.284 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-01-25T06:30:37.285 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:37.285 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:37.285 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.22991 2023-01-25T06:30:37.287 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-01-25T06:30:37.288 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-01-25T06:30:37.289 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-01-25T06:30:37.289 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/balancer 1' TERM HUP INT 2023-01-25T06:30:37.289 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:136: TEST_balancer2: run_mon td/balancer a 2023-01-25T06:30:37.289 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/balancer 2023-01-25T06:30:37.289 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-01-25T06:30:37.290 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-01-25T06:30:37.290 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-01-25T06:30:37.290 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/balancer/a 2023-01-25T06:30:37.290 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/balancer/a --run-dir=td/balancer 2023-01-25T06:30:37.683 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-01-25T06:30:37.683 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:30:37.684 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:30:37.684 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:30:37.685 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:37.685 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:37.685 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:30:37.686 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/balancer/a '--log-file=td/balancer/$name.log' '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --mon-cluster-log-file=td/balancer/log --run-dir=td/balancer '--pid-file=td/balancer/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-01-25T06:30:37.974 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-01-25T06:30:37.976 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-01-25T06:30:37.976 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T06:30:37.976 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T06:30:37.977 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=fsid 2023-01-25T06:30:37.978 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .fsid 2023-01-25T06:30:37.978 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T06:30:37.978 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T06:30:37.978 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T06:30:37.979 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T06:30:37.979 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:37.979 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:37.979 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.22991/ceph-mon.a.asok 2023-01-25T06:30:37.980 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T06:30:37.980 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.22991/ceph-mon.a.asok config get fsid 2023-01-25T06:30:38.076 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-01-25T06:30:38.076 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: local daemon=mon 2023-01-25T06:30:38.076 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1124: get_config: local id=a 2023-01-25T06:30:38.076 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local config=mon_host 2023-01-25T06:30:38.077 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: get_config: jq -r .mon_host 2023-01-25T06:30:38.077 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: get_asok_path mon.a 2023-01-25T06:30:38.078 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-01-25T06:30:38.078 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-01-25T06:30:38.079 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-01-25T06:30:38.079 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:38.079 INFO:tasks.workunit.client.0.smithi040.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:38.079 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.22991/ceph-mon.a.asok 2023-01-25T06:30:38.080 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: CEPH_ARGS= 2023-01-25T06:30:38.080 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: ceph --format json daemon /tmp/ceph-asok.22991/ceph-mon.a.asok config get mon_host 2023-01-25T06:30:38.169 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:137: TEST_balancer2: run_mgr td/balancer x 2023-01-25T06:30:38.169 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/balancer 2023-01-25T06:30:38.169 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-01-25T06:30:38.170 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-01-25T06:30:38.170 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-01-25T06:30:38.170 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/balancer/x 2023-01-25T06:30:38.170 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-01-25T06:30:39.071 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-01-25T06:30:39.072 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:30:39.072 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:30:39.073 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:30:39.073 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:39.073 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:39.073 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:30:39.074 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T06:30:39.075 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/balancer/x '--log-file=td/balancer/$name.log' '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --run-dir=td/balancer '--pid-file=td/balancer/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-01-25T06:30:39.142 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: expr 5 - 1 2023-01-25T06:30:39.143 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: seq 0 4 2023-01-25T06:30:39.172 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: for i in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T06:30:39.172 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:140: TEST_balancer2: run_osd td/balancer 0 2023-01-25T06:30:39.172 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:30:39.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:30:39.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-01-25T06:30:39.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:30:39.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/0 2023-01-25T06:30:39.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 ' 2023-01-25T06:30:39.174 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:30:39.174 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:30:39.174 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:30:39.174 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/0' 2023-01-25T06:30:39.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/0/journal' 2023-01-25T06:30:39.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:30:39.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:30:39.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:30:39.175 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:30:39.176 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:30:39.176 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:30:39.178 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:30:39.178 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:30:39.178 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:30:39.178 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:30:39.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:30:39.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:30:39.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:30:39.179 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:30:39.180 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:30:39.180 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:30:39.180 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:30:39.180 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:30:39.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:30:39.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:30:39.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:30:39.181 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/0 2023-01-25T06:30:39.181 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:30:39.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=eba940d7-df49-41c9-adff-5edcb21582e4 2023-01-25T06:30:39.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 eba940d7-df49-41c9-adff-5edcb21582e4' 2023-01-25T06:30:39.183 INFO:tasks.workunit.client.0.smithi040.stdout:add osd0 eba940d7-df49-41c9-adff-5edcb21582e4 2023-01-25T06:30:39.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:30:39.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCPzNBjoi4vDBAAK1KqM+93aTOqNEcXg8z2Gg== 2023-01-25T06:30:39.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCPzNBjoi4vDBAAK1KqM+93aTOqNEcXg8z2Gg=="}' 2023-01-25T06:30:39.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new eba940d7-df49-41c9-adff-5edcb21582e4 -i td/balancer/0/new.json 2023-01-25T06:30:39.565 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:30:39.580 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/0/new.json 2023-01-25T06:30:39.582 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/0 --osd-journal=td/balancer/0/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCPzNBjoi4vDBAAK1KqM+93aTOqNEcXg8z2Gg== --osd-uuid eba940d7-df49-41c9-adff-5edcb21582e4 2023-01-25T06:30:39.617 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.613+0000 7f0178915680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:39.619 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.617+0000 7f0178915680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:39.622 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.617+0000 7f0178915680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:39.622 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.617+0000 7f0178915680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:30:39.623 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.617+0000 7f0178915680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:30:39.623 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.617+0000 7f0178915680 -1 bluestore(td/balancer/0/block) _read_bdev_label failed to open td/balancer/0/block: (2) No such file or directory 2023-01-25T06:30:39.713 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:39.709+0000 7f0178915680 -1 bluestore(td/balancer/0) _read_fsid unparsable uuid 2023-01-25T06:30:45.780 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/0/keyring 2023-01-25T06:30:45.780 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:30:45.781 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd0 key to auth repository 2023-01-25T06:30:45.782 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2023-01-25T06:30:45.782 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:30:46.330 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.0 2023-01-25T06:30:46.331 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2023-01-25T06:30:46.331 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/0 --osd-journal=td/balancer/0/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:30:46.331 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:30:46.332 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:30:46.332 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:30:46.366 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:46.361+0000 7f8f32a9f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:46.380 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:46.377+0000 7f8f32a9f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:46.382 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:46.377+0000 7f8f32a9f680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:30:46.806 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2023-01-25T06:30:46.806 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:30:46.806 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=0 2023-01-25T06:30:46.806 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:30:46.807 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:30:46.807 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:46.807 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:30:46.807 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:30:46.808 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:46.808 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:46.954 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:46.949+0000 7f8f32a9f680 -1 Falling back to public interface 2023-01-25T06:30:47.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:48.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:48.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:48.207 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:30:48.208 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:30:48.208 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:48.209 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:48.627 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:49.557 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:30:49.553+0000 7f8f32a9f680 -1 osd.0 0 log_to_monitors true 2023-01-25T06:30:49.629 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:49.630 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:49.630 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:30:49.631 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:30:49.631 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:49.631 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:50.148 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:51.150 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:51.151 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:51.151 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:30:51.151 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:30:51.152 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:51.152 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:51.580 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:52.581 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:52.581 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:52.582 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:30:52.582 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:30:52.583 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:52.583 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:53.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:54.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:54.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:54.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:30:54.011 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:30:54.011 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:54.012 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:54.542 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:55.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:55.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:55.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:30:55.546 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:30:55.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:55.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:55.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:56.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:56.963 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:56.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:30:56.964 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:30:56.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:56.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:57.409 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:58.411 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:58.412 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:30:58.412 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:30:58.413 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:58.413 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:30:58.864 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:30:59.866 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:30:59.867 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:30:59.867 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:30:59.867 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:30:59.868 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:30:59.868 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:00.390 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:01.392 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:01.393 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:01.393 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:31:01.394 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:31:01.394 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:01.394 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:02.046 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:03.048 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:03.049 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:03.049 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:31:03.049 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:31:03.050 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:03.050 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:03.656 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:04.658 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:04.658 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:04.658 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:31:04.659 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:31:04.659 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:04.659 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:05.410 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:06.411 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:06.411 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:06.412 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:31:06.412 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:31:06.412 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:06.413 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:07.085 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:08.087 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:08.088 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:08.088 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:31:08.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:31:08.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:08.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:08.701 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:09.703 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:09.703 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:09.704 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:31:09.704 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:31:09.705 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:09.705 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:10.501 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:11.503 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:11.503 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:11.504 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:31:11.505 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:31:11.505 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:11.505 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.0 up' 2023-01-25T06:31:11.939 INFO:tasks.workunit.client.0.smithi040.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31189,v1:127.0.0.1:6803/31189] [v2:127.0.0.1:6804/31189,v1:127.0.0.1:6805/31189] exists,up eba940d7-df49-41c9-adff-5edcb21582e4 2023-01-25T06:31:11.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:31:11.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:31:11.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:31:11.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: for i in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T06:31:11.944 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:140: TEST_balancer2: run_osd td/balancer 1 2023-01-25T06:31:11.944 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:31:11.944 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:31:11.944 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-01-25T06:31:11.944 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:31:11.945 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/1 2023-01-25T06:31:11.945 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 ' 2023-01-25T06:31:11.945 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:31:11.945 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:31:11.945 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:31:11.946 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/1' 2023-01-25T06:31:11.946 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/1/journal' 2023-01-25T06:31:11.946 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:31:11.946 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:31:11.946 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:31:11.947 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:31:11.947 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:31:11.947 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:31:11.947 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:31:11.947 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:31:11.948 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:31:11.948 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:31:11.948 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:31:11.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:31:11.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:31:11.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:31:11.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:31:11.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:31:11.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:31:11.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:31:11.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:31:11.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:31:11.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:31:11.951 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/1 2023-01-25T06:31:11.963 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:31:11.964 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e 2023-01-25T06:31:11.965 INFO:tasks.workunit.client.0.smithi040.stdout:add osd1 66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e 2023-01-25T06:31:11.965 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e' 2023-01-25T06:31:11.965 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:31:11.991 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCvzNBjhFn+OhAAABtjG3nnFt4YhzNRKrQyJw== 2023-01-25T06:31:11.991 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCvzNBjhFn+OhAAABtjG3nnFt4YhzNRKrQyJw=="}' 2023-01-25T06:31:11.991 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e -i td/balancer/1/new.json 2023-01-25T06:31:12.506 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:31:12.520 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/1/new.json 2023-01-25T06:31:12.522 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/1 --osd-journal=td/balancer/1/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCvzNBjhFn+OhAAABtjG3nnFt4YhzNRKrQyJw== --osd-uuid 66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e 2023-01-25T06:31:12.558 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.552+0000 7f4a5c8d3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:12.587 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.580+0000 7f4a5c8d3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:12.589 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.584+0000 7f4a5c8d3680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:12.591 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.584+0000 7f4a5c8d3680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:31:12.591 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.584+0000 7f4a5c8d3680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:31:12.592 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.584+0000 7f4a5c8d3680 -1 bluestore(td/balancer/1/block) _read_bdev_label failed to open td/balancer/1/block: (2) No such file or directory 2023-01-25T06:31:12.721 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:12.720+0000 7f4a5c8d3680 -1 bluestore(td/balancer/1) _read_fsid unparsable uuid 2023-01-25T06:31:18.893 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/1/keyring 2023-01-25T06:31:18.893 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:31:18.895 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2023-01-25T06:31:18.895 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd1 key to auth repository 2023-01-25T06:31:18.896 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:31:19.445 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2023-01-25T06:31:19.446 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.1 2023-01-25T06:31:19.447 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/1 --osd-journal=td/balancer/1/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:31:19.447 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:31:19.447 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:31:19.447 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:31:19.481 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:19.476+0000 7f4d97244680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:19.484 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:19.480+0000 7f4d97244680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:19.487 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:19.480+0000 7f4d97244680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:19.859 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2023-01-25T06:31:19.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:31:19.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=1 2023-01-25T06:31:19.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:31:19.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:31:19.861 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:19.861 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:31:19.862 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:31:19.862 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:19.862 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:20.046 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:20.040+0000 7f4d97244680 -1 Falling back to public interface 2023-01-25T06:31:20.270 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:21.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:21.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:21.273 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:31:21.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:31:21.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:21.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:21.687 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:22.548 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:22.544+0000 7f4d97244680 -1 osd.1 0 log_to_monitors true 2023-01-25T06:31:22.689 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:22.689 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:22.689 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:31:22.690 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:31:22.690 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:22.690 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:23.336 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:24.338 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:24.338 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:24.338 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:31:24.338 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:31:24.339 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:24.339 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:24.808 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:25.809 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:25.810 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:25.810 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:31:25.811 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:31:25.811 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:25.811 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:26.322 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:27.324 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:27.324 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:27.325 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:31:27.326 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:31:27.326 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:27.326 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:27.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:28.777 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:28.778 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:28.778 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:31:28.778 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:31:28.779 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:28.779 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:29.398 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:30.399 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:30.400 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:30.400 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:31:30.401 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:31:30.401 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:30.401 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:30.837 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:31.838 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:31.839 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:31.839 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:31:31.840 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:31:31.840 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:31.840 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:32.265 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:33.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:33.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:33.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:31:33.268 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:31:33.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:33.269 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:33.722 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:34.723 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:34.724 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:34.724 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:31:34.725 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:31:34.725 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:34.725 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:35.647 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:36.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:36.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:36.650 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:31:36.650 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:31:36.650 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:36.651 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:37.357 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:38.359 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:38.360 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:38.360 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:31:38.361 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:31:38.361 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:38.361 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:39.240 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:40.242 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:40.243 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:40.243 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:31:40.244 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:31:40.244 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:40.244 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:40.975 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:41.977 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:41.978 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:41.978 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:31:41.978 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:31:41.979 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:41.979 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:42.726 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:43.728 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:43.729 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:43.729 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:31:43.730 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:31:43.730 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:43.730 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:44.772 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:45.774 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:45.774 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:45.774 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:31:45.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:31:45.775 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:45.776 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.1 up' 2023-01-25T06:31:46.201 INFO:tasks.workunit.client.0.smithi040.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32922,v1:127.0.0.1:6811/32922] [v2:127.0.0.1:6812/32922,v1:127.0.0.1:6813/32922] exists,up 66dde8d9-d7d8-40d7-acc0-e8ce1e37ae2e 2023-01-25T06:31:46.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:31:46.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:31:46.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:31:46.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: for i in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T06:31:46.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:140: TEST_balancer2: run_osd td/balancer 2 2023-01-25T06:31:46.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:31:46.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:31:46.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-01-25T06:31:46.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:31:46.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/2 2023-01-25T06:31:46.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 ' 2023-01-25T06:31:46.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:31:46.208 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:31:46.208 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:31:46.209 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/2' 2023-01-25T06:31:46.209 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/2/journal' 2023-01-25T06:31:46.210 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:31:46.210 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:31:46.210 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:31:46.211 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:31:46.211 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:31:46.211 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:31:46.212 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:31:46.212 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:31:46.213 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:31:46.216 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:31:46.217 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:31:46.217 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:31:46.217 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:31:46.218 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:31:46.218 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:31:46.218 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:31:46.219 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:31:46.219 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:31:46.220 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:31:46.220 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:31:46.220 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:31:46.221 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/2 2023-01-25T06:31:46.221 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:31:46.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=221d757d-224a-44a7-a366-9cd160ed0731 2023-01-25T06:31:46.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 221d757d-224a-44a7-a366-9cd160ed0731' 2023-01-25T06:31:46.223 INFO:tasks.workunit.client.0.smithi040.stdout:add osd2 221d757d-224a-44a7-a366-9cd160ed0731 2023-01-25T06:31:46.223 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:31:46.230 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDSzNBjXASlDRAA2Ug4atzmV67glslfXSMxFA== 2023-01-25T06:31:46.231 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDSzNBjXASlDRAA2Ug4atzmV67glslfXSMxFA=="}' 2023-01-25T06:31:46.231 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 221d757d-224a-44a7-a366-9cd160ed0731 -i td/balancer/2/new.json 2023-01-25T06:31:46.750 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:31:46.762 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/2/new.json 2023-01-25T06:31:46.764 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/2 --osd-journal=td/balancer/2/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSzNBjXASlDRAA2Ug4atzmV67glslfXSMxFA== --osd-uuid 221d757d-224a-44a7-a366-9cd160ed0731 2023-01-25T06:31:46.803 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.796+0000 7f5ae934a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:46.817 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.812+0000 7f5ae934a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:46.819 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.812+0000 7f5ae934a680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:46.820 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.816+0000 7f5ae934a680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:31:46.820 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.816+0000 7f5ae934a680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:31:46.820 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.816+0000 7f5ae934a680 -1 bluestore(td/balancer/2/block) _read_bdev_label failed to open td/balancer/2/block: (2) No such file or directory 2023-01-25T06:31:46.862 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:46.856+0000 7f5ae934a680 -1 bluestore(td/balancer/2) _read_fsid unparsable uuid 2023-01-25T06:31:53.124 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/2/keyring 2023-01-25T06:31:53.124 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:31:53.126 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd2 key to auth repository 2023-01-25T06:31:53.126 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2023-01-25T06:31:53.126 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:31:53.699 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.2 2023-01-25T06:31:53.700 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2023-01-25T06:31:53.700 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/2 --osd-journal=td/balancer/2/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:31:53.700 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:31:53.701 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:31:53.701 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:31:53.735 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:53.727+0000 7f9578ec5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:53.754 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:53.747+0000 7f9578ec5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:53.756 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:53.747+0000 7f9578ec5680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:31:54.097 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2023-01-25T06:31:54.097 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:31:54.097 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=2 2023-01-25T06:31:54.097 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:31:54.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:31:54.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:54.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:31:54.098 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:31:54.099 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:54.099 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:31:54.318 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:54.311+0000 7f9578ec5680 -1 Falling back to public interface 2023-01-25T06:31:54.594 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:55.596 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:55.597 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:55.597 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:31:55.597 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:31:55.598 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:55.598 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:31:56.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:56.673 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:31:56.667+0000 7f9578ec5680 -1 osd.2 0 log_to_monitors true 2023-01-25T06:31:57.040 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:57.040 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:57.040 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:31:57.040 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:31:57.041 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:57.041 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:31:57.525 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:58.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:58.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:58.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:31:58.528 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:31:58.528 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:58.529 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:31:58.971 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:31:59.973 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:31:59.973 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:31:59.974 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:31:59.975 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:31:59.975 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:31:59.976 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:00.584 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:01.586 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:01.586 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:01.587 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:32:01.587 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:32:01.588 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:01.588 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:02.019 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:03.021 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:03.021 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:03.021 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:32:03.022 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:32:03.022 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:03.022 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:03.499 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:04.501 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:04.502 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:04.502 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:32:04.503 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:32:04.503 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:04.503 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:04.996 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:05.998 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:05.998 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:05.998 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:32:05.999 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:32:05.999 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:05.999 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:06.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:07.569 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:07.570 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:07.570 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:32:07.571 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:32:07.571 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:07.571 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:07.994 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:08.996 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:08.996 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:08.997 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:32:08.997 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:32:08.998 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:08.998 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:09.597 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:10.599 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:10.599 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:10.600 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:32:10.601 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:32:10.601 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:10.601 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:11.306 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:12.308 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:12.308 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:12.308 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:32:12.309 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:32:12.309 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:12.309 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:12.985 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:13.987 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:13.988 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:13.988 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:32:13.989 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:32:13.989 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:13.989 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:14.849 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:15.851 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:15.851 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:15.851 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:32:15.852 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:32:15.852 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:15.853 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:16.859 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:17.861 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:17.862 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:17.862 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:32:17.862 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:32:17.863 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:17.863 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:18.690 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:19.692 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:19.693 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:19.693 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:32:19.693 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:32:19.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:19.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.2 up' 2023-01-25T06:32:20.194 INFO:tasks.workunit.client.0.smithi040.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/34655,v1:127.0.0.1:6819/34655] [v2:127.0.0.1:6820/34655,v1:127.0.0.1:6821/34655] exists,up 221d757d-224a-44a7-a366-9cd160ed0731 2023-01-25T06:32:20.198 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:32:20.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:32:20.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:32:20.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: for i in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T06:32:20.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:140: TEST_balancer2: run_osd td/balancer 3 2023-01-25T06:32:20.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:32:20.200 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:32:20.200 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-01-25T06:32:20.200 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:32:20.200 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/3 2023-01-25T06:32:20.200 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 ' 2023-01-25T06:32:20.201 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:32:20.201 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:32:20.201 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:32:20.201 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/3' 2023-01-25T06:32:20.201 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/3/journal' 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:32:20.202 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:32:20.203 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:32:20.203 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:32:20.203 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:32:20.203 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:32:20.203 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:32:20.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:32:20.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:32:20.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:32:20.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:32:20.204 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:32:20.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:32:20.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:32:20.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:32:20.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:32:20.205 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:32:20.206 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/3 2023-01-25T06:32:20.206 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:32:20.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=6f616ddb-9b11-45c4-b440-ffd3fc9656da 2023-01-25T06:32:20.207 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 6f616ddb-9b11-45c4-b440-ffd3fc9656da' 2023-01-25T06:32:20.207 INFO:tasks.workunit.client.0.smithi040.stdout:add osd3 6f616ddb-9b11-45c4-b440-ffd3fc9656da 2023-01-25T06:32:20.207 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:32:20.229 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD0zNBjDRmJDRAA0LhFCWpueglhAz4S2zV1CA== 2023-01-25T06:32:20.229 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD0zNBjDRmJDRAA0LhFCWpueglhAz4S2zV1CA=="}' 2023-01-25T06:32:20.229 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 6f616ddb-9b11-45c4-b440-ffd3fc9656da -i td/balancer/3/new.json 2023-01-25T06:32:20.743 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:32:20.754 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/3/new.json 2023-01-25T06:32:20.756 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/3 --osd-journal=td/balancer/3/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD0zNBjDRmJDRAA0LhFCWpueglhAz4S2zV1CA== --osd-uuid 6f616ddb-9b11-45c4-b440-ffd3fc9656da 2023-01-25T06:32:20.792 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.787+0000 7f486d762680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:20.795 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.791+0000 7f486d762680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:20.797 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.795+0000 7f486d762680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:20.798 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.795+0000 7f486d762680 -1 bluestore(td/balancer/3/block) _read_bdev_label failed to open td/balancer/3/block: (2) No such file or directory 2023-01-25T06:32:20.798 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.795+0000 7f486d762680 -1 bluestore(td/balancer/3/block) _read_bdev_label failed to open td/balancer/3/block: (2) No such file or directory 2023-01-25T06:32:20.798 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.795+0000 7f486d762680 -1 bluestore(td/balancer/3/block) _read_bdev_label failed to open td/balancer/3/block: (2) No such file or directory 2023-01-25T06:32:20.874 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:20.871+0000 7f486d762680 -1 bluestore(td/balancer/3) _read_fsid unparsable uuid 2023-01-25T06:32:27.124 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/3/keyring 2023-01-25T06:32:27.124 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:32:27.126 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2023-01-25T06:32:27.126 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd3 key to auth repository 2023-01-25T06:32:27.127 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:32:27.678 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.3 2023-01-25T06:32:27.679 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2023-01-25T06:32:27.679 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/3 --osd-journal=td/balancer/3/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:32:27.679 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:32:27.680 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:32:27.680 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:32:27.715 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:27.711+0000 7f13e1e6d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:27.733 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:27.731+0000 7f13e1e6d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:27.735 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:27.731+0000 7f13e1e6d680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:28.117 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2023-01-25T06:32:28.117 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:32:28.117 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=3 2023-01-25T06:32:28.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:32:28.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:32:28.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:28.118 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:32:28.119 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:32:28.119 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:28.119 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:28.318 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:28.315+0000 7f13e1e6d680 -1 Falling back to public interface 2023-01-25T06:32:28.558 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:29.560 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:29.561 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:29.561 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:32:29.562 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:32:29.562 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:29.563 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:29.993 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:30.809 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:30.807+0000 7f13e1e6d680 -1 osd.3 0 log_to_monitors true 2023-01-25T06:32:30.994 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:30.995 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:30.995 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:32:30.996 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:32:30.997 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:30.997 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:31.565 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:32.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:32.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:32.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:32:32.568 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:32:32.568 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:32.568 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:33.008 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:34.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:34.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:34.011 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:32:34.011 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:32:34.011 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:34.012 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:34.512 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:35.513 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:35.514 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:35.514 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:32:35.515 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:32:35.515 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:35.515 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:35.950 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:36.952 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:36.953 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:36.953 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:32:36.954 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:32:36.954 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:36.955 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:37.526 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:38.528 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:38.529 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:38.529 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:32:38.530 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:32:38.530 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:38.530 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:38.957 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:39.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:39.960 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:39.960 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:32:39.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:32:39.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:39.961 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:40.382 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:41.383 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:41.384 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:41.384 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:32:41.384 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:32:41.385 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:41.385 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:42.009 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:43.010 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:43.011 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:43.011 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:32:43.012 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:32:43.012 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:43.012 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:43.646 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:44.648 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:44.648 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:44.648 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:32:44.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:32:44.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:44.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:45.439 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:46.441 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:46.441 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:46.441 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:32:46.442 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:32:46.442 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:46.442 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:47.096 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:48.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:48.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:48.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:32:48.099 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:32:48.099 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:48.100 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:49.195 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:50.197 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:50.197 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:50.198 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:32:50.198 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:32:50.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:50.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:51.138 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:52.141 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:52.141 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:52.141 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:32:52.142 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:32:52.142 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:52.143 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:52.757 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:32:53.759 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:32:53.760 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:32:53.760 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:32:53.760 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:32:53.761 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:32:53.761 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.3 up' 2023-01-25T06:32:54.268 INFO:tasks.workunit.client.0.smithi040.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/36388,v1:127.0.0.1:6827/36388] [v2:127.0.0.1:6828/36388,v1:127.0.0.1:6829/36388] exists,up 6f616ddb-9b11-45c4-b440-ffd3fc9656da 2023-01-25T06:32:54.270 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:32:54.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:32:54.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:32:54.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:138: TEST_balancer2: for i in $(seq 0 $(expr $OSDS - 1)) 2023-01-25T06:32:54.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:140: TEST_balancer2: run_osd td/balancer 4 2023-01-25T06:32:54.271 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/balancer 2023-01-25T06:32:54.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-01-25T06:32:54.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-01-25T06:32:54.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-01-25T06:32:54.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/balancer/4 2023-01-25T06:32:54.272 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 ' 2023-01-25T06:32:54.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-01-25T06:32:54.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-01-25T06:32:54.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-01-25T06:32:54.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/balancer/4' 2023-01-25T06:32:54.273 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/balancer/4/journal' 2023-01-25T06:32:54.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-01-25T06:32:54.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-01-25T06:32:54.274 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/balancer' 2023-01-25T06:32:54.274 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-01-25T06:32:54.274 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-01-25T06:32:54.275 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-01-25T06:32:54.275 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-01-25T06:32:54.277 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:32:54.277 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:32:54.277 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:32:54.277 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' 2023-01-25T06:32:54.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-01-25T06:32:54.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-01-25T06:32:54.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-01-25T06:32:54.278 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/balancer/$name.log' 2023-01-25T06:32:54.279 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/balancer/$name.pid' 2023-01-25T06:32:54.279 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-01-25T06:32:54.279 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-01-25T06:32:54.279 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:32:54.279 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2023-01-25T06:32:54.280 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2023-01-25T06:32:54.280 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/balancer/4 2023-01-25T06:32:54.280 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2023-01-25T06:32:54.280 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5b68b161-801f-41ca-8a99-489149961b46 2023-01-25T06:32:54.280 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 5b68b161-801f-41ca-8a99-489149961b46' 2023-01-25T06:32:54.281 INFO:tasks.workunit.client.0.smithi040.stdout:add osd4 5b68b161-801f-41ca-8a99-489149961b46 2023-01-25T06:32:54.281 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2023-01-25T06:32:54.305 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAWzdBj0IALEhAAtjF4+gzi8m1Rrh6Zv33VCQ== 2023-01-25T06:32:54.305 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAWzdBj0IALEhAAtjF4+gzi8m1Rrh6Zv33VCQ=="}' 2023-01-25T06:32:54.306 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5b68b161-801f-41ca-8a99-489149961b46 -i td/balancer/4/new.json 2023-01-25T06:32:54.824 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:32:54.841 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/balancer/4/new.json 2023-01-25T06:32:54.843 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/4 --osd-journal=td/balancer/4/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAWzdBj0IALEhAAtjF4+gzi8m1Rrh6Zv33VCQ== --osd-uuid 5b68b161-801f-41ca-8a99-489149961b46 2023-01-25T06:32:54.879 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.874+0000 7fb6121d1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:54.885 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.882+0000 7fb6121d1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:54.886 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.882+0000 7fb6121d1680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:32:54.888 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.882+0000 7fb6121d1680 -1 bluestore(td/balancer/4/block) _read_bdev_label failed to open td/balancer/4/block: (2) No such file or directory 2023-01-25T06:32:54.888 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.882+0000 7fb6121d1680 -1 bluestore(td/balancer/4/block) _read_bdev_label failed to open td/balancer/4/block: (2) No such file or directory 2023-01-25T06:32:54.888 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.882+0000 7fb6121d1680 -1 bluestore(td/balancer/4/block) _read_bdev_label failed to open td/balancer/4/block: (2) No such file or directory 2023-01-25T06:32:54.931 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:32:54.926+0000 7fb6121d1680 -1 bluestore(td/balancer/4) _read_fsid unparsable uuid 2023-01-25T06:33:01.453 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/balancer/4/keyring 2023-01-25T06:33:01.454 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2023-01-25T06:33:01.456 INFO:tasks.workunit.client.0.smithi040.stdout:adding osd4 key to auth repository 2023-01-25T06:33:01.456 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2023-01-25T06:33:01.457 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/balancer/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-01-25T06:33:02.013 INFO:tasks.workunit.client.0.smithi040.stdout:start osd.4 2023-01-25T06:33:02.015 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2023-01-25T06:33:02.015 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=a1aed31b-9db7-4f94-9ef9-c8ddb06d3ef0 --auth-supported=none --mon-host=127.0.0.1:7102 --osd_pool_default_pg_autoscale_mode=off --debug_osd=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/balancer/4 --osd-journal=td/balancer/4/journal --chdir= --run-dir=td/balancer '--admin-socket=/tmp/ceph-asok.22991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/balancer/$name.log' '--pid-file=td/balancer/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2023-01-25T06:33:02.016 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2023-01-25T06:33:02.016 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2023-01-25T06:33:02.016 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2023-01-25T06:33:02.048 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:33:02.042+0000 7fa1519a0680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:33:02.076 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:33:02.070+0000 7fa1519a0680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:33:02.080 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:33:02.074+0000 7fa1519a0680 -1 WARNING: all dangerous and experimental features are enabled. 2023-01-25T06:33:02.507 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2023-01-25T06:33:02.507 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: local state=up 2023-01-25T06:33:02.507 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:977: wait_for_osd: local id=4 2023-01-25T06:33:02.508 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: status=1 2023-01-25T06:33:02.508 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i=0 )) 2023-01-25T06:33:02.509 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:02.510 INFO:tasks.workunit.client.0.smithi040.stdout:0 2023-01-25T06:33:02.510 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 0 2023-01-25T06:33:02.511 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:02.511 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:02.642 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:33:02.638+0000 7fa1519a0680 -1 Falling back to public interface 2023-01-25T06:33:03.039 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:04.041 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:04.042 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:04.042 INFO:tasks.workunit.client.0.smithi040.stdout:1 2023-01-25T06:33:04.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 1 2023-01-25T06:33:04.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:04.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:04.517 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:05.225 INFO:tasks.workunit.client.0.smithi040.stderr:2023-01-25T06:33:05.222+0000 7fa1519a0680 -1 osd.4 0 log_to_monitors true 2023-01-25T06:33:05.519 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:05.519 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:05.519 INFO:tasks.workunit.client.0.smithi040.stdout:2 2023-01-25T06:33:05.520 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 2 2023-01-25T06:33:05.520 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:05.521 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:06.035 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:07.037 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:07.037 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:07.038 INFO:tasks.workunit.client.0.smithi040.stdout:3 2023-01-25T06:33:07.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 3 2023-01-25T06:33:07.038 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:07.039 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:07.606 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:08.607 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:08.608 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:08.608 INFO:tasks.workunit.client.0.smithi040.stdout:4 2023-01-25T06:33:08.609 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 4 2023-01-25T06:33:08.609 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:08.609 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:09.104 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:10.106 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:10.106 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:10.106 INFO:tasks.workunit.client.0.smithi040.stdout:5 2023-01-25T06:33:10.107 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 5 2023-01-25T06:33:10.107 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:10.107 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:10.562 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:11.564 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:11.565 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:11.566 INFO:tasks.workunit.client.0.smithi040.stdout:6 2023-01-25T06:33:11.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 6 2023-01-25T06:33:11.568 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:11.568 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:12.147 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:13.149 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:13.150 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:13.150 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 7 2023-01-25T06:33:13.150 INFO:tasks.workunit.client.0.smithi040.stdout:7 2023-01-25T06:33:13.151 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:13.151 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:13.645 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:14.647 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:14.648 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:14.648 INFO:tasks.workunit.client.0.smithi040.stdout:8 2023-01-25T06:33:14.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 8 2023-01-25T06:33:14.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:14.649 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:15.098 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:16.100 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:16.100 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:16.101 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 9 2023-01-25T06:33:16.101 INFO:tasks.workunit.client.0.smithi040.stdout:9 2023-01-25T06:33:16.101 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:16.102 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:16.543 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:17.544 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:17.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:17.545 INFO:tasks.workunit.client.0.smithi040.stdout:10 2023-01-25T06:33:17.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 10 2023-01-25T06:33:17.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:17.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:18.171 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:19.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:19.173 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:19.174 INFO:tasks.workunit.client.0.smithi040.stdout:11 2023-01-25T06:33:19.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 11 2023-01-25T06:33:19.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:19.175 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:19.709 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:20.711 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:20.712 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:20.713 INFO:tasks.workunit.client.0.smithi040.stdout:12 2023-01-25T06:33:20.713 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 12 2023-01-25T06:33:20.713 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:20.714 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:21.573 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:22.575 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:22.575 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:22.576 INFO:tasks.workunit.client.0.smithi040.stdout:13 2023-01-25T06:33:22.576 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 13 2023-01-25T06:33:22.577 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:22.577 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:23.664 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:24.666 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:24.667 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:24.667 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 14 2023-01-25T06:33:24.667 INFO:tasks.workunit.client.0.smithi040.stdout:14 2023-01-25T06:33:24.668 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:24.668 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:25.733 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:26.735 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:26.735 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:26.736 INFO:tasks.workunit.client.0.smithi040.stdout:15 2023-01-25T06:33:26.736 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 15 2023-01-25T06:33:26.736 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:26.737 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:27.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: sleep 1 2023-01-25T06:33:28.547 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i++ )) 2023-01-25T06:33:28.547 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:980: wait_for_osd: (( i < 300 )) 2023-01-25T06:33:28.548 INFO:tasks.workunit.client.0.smithi040.stdout:16 2023-01-25T06:33:28.548 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: echo 16 2023-01-25T06:33:28.549 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: ceph osd dump 2023-01-25T06:33:28.549 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: grep 'osd.4 up' 2023-01-25T06:33:29.088 INFO:tasks.workunit.client.0.smithi040.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/38131,v1:127.0.0.1:6835/38131] [v2:127.0.0.1:6836/38131,v1:127.0.0.1:6837/38131] exists,up 5b68b161-801f-41ca-8a99-489149961b46 2023-01-25T06:33:29.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: status=0 2023-01-25T06:33:29.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: wait_for_osd: break 2023-01-25T06:33:29.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: wait_for_osd: return 0 2023-01-25T06:33:29.089 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:143: TEST_balancer2: ceph osd set-require-min-compat-client luminous 2023-01-25T06:33:29.814 INFO:tasks.workunit.client.0.smithi040.stderr:set require_min_compat_client to luminous 2023-01-25T06:33:29.827 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:144: TEST_balancer2: ceph config set mgr mgr/balancer/upmap_max_deviation 1 2023-01-25T06:33:30.427 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:145: TEST_balancer2: ceph balancer mode upmap 2023-01-25T06:33:30.958 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:146: TEST_balancer2: ceph balancer on 2023-01-25T06:33:31.372 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:147: TEST_balancer2: ceph config set mgr mgr/balancer/sleep_interval 5 2023-01-25T06:33:31.875 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:149: TEST_balancer2: create_pool test1 118 2023-01-25T06:33:31.876 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 118 2023-01-25T06:33:32.469 INFO:tasks.workunit.client.0.smithi040.stderr:pool 'test1' already exists 2023-01-25T06:33:32.479 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T06:33:33.481 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:151: TEST_balancer2: wait_for_clean 2023-01-25T06:33:33.481 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T06:33:33.481 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T06:33:33.482 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T06:33:33.482 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T06:33:33.483 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T06:33:33.483 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T06:33:33.483 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T06:33:33.484 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T06:33:33.484 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T06:33:33.484 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T06:33:33.661 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T06:33:33.661 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T06:33:33.661 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T06:33:33.661 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T06:33:33.662 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T06:33:34.364 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T06:33:34.364 INFO:tasks.workunit.client.0.smithi040.stderr:1 2023-01-25T06:33:34.365 INFO:tasks.workunit.client.0.smithi040.stderr:2 2023-01-25T06:33:34.365 INFO:tasks.workunit.client.0.smithi040.stderr:3 2023-01-25T06:33:34.365 INFO:tasks.workunit.client.0.smithi040.stderr:4' 2023-01-25T06:33:34.365 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T06:33:34.365 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:33:34.366 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T06:33:34.566 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836518 2023-01-25T06:33:34.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836518 2023-01-25T06:33:34.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836518' 2023-01-25T06:33:34.567 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:33:34.568 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T06:33:34.728 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949672994 2023-01-25T06:33:34.728 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949672994 2023-01-25T06:33:34.728 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836518 1-42949672994' 2023-01-25T06:33:34.729 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:33:34.729 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T06:33:34.882 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509465 2023-01-25T06:33:34.883 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509465 2023-01-25T06:33:34.883 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836518 1-42949672994 2-64424509465' 2023-01-25T06:33:34.883 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:33:34.883 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T06:33:35.066 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345937 2023-01-25T06:33:35.066 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345937 2023-01-25T06:33:35.066 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836518 1-42949672994 2-64424509465 3-85899345937' 2023-01-25T06:33:35.066 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:33:35.067 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-25T06:33:35.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182406 2023-01-25T06:33:35.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182406 2023-01-25T06:33:35.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836518 1-42949672994 2-64424509465 3-85899345937 4-107374182406' 2023-01-25T06:33:35.233 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:33:35.234 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836518 2023-01-25T06:33:35.234 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:33:35.235 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T06:33:35.236 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836518 2023-01-25T06:33:35.237 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:33:35.239 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836518 2023-01-25T06:33:35.239 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2023-01-25T06:33:35.239 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.0 seq 21474836518 2023-01-25T06:33:35.240 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:33:35.760 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836516 -lt 21474836518 2023-01-25T06:33:35.760 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T06:33:36.761 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T06:33:36.762 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:33:37.184 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836518 -lt 21474836518 2023-01-25T06:33:37.184 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:33:37.184 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949672994 2023-01-25T06:33:37.185 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:33:37.187 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T06:33:37.188 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949672994 2023-01-25T06:33:37.188 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:33:37.190 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949672994 2023-01-25T06:33:37.190 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2023-01-25T06:33:37.190 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.1 seq 42949672994 2023-01-25T06:33:37.191 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T06:33:37.620 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949672995 -lt 42949672994 2023-01-25T06:33:37.620 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:33:37.621 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509465 2023-01-25T06:33:37.622 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:33:37.622 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T06:33:37.624 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509465 2023-01-25T06:33:37.624 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:33:37.626 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509465 2023-01-25T06:33:37.626 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2023-01-25T06:33:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.2 seq 64424509465 2023-01-25T06:33:37.627 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T06:33:38.046 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509465 -lt 64424509465 2023-01-25T06:33:38.047 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:33:38.048 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345937 2023-01-25T06:33:38.048 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:33:38.050 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T06:33:38.051 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345937 2023-01-25T06:33:38.051 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:33:38.053 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345937 2023-01-25T06:33:38.053 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2023-01-25T06:33:38.053 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.3 seq 85899345937 2023-01-25T06:33:38.054 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T06:33:38.472 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345937 -lt 85899345937 2023-01-25T06:33:38.472 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:33:38.473 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182406 2023-01-25T06:33:38.473 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:33:38.474 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-25T06:33:38.475 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182406 2023-01-25T06:33:38.476 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:33:38.477 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182406 2023-01-25T06:33:38.478 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.4 seq 107374182406 2023-01-25T06:33:38.479 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2023-01-25T06:33:38.479 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-25T06:33:38.906 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182406 -lt 107374182406 2023-01-25T06:33:38.906 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T06:33:38.907 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:33:38.907 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:33:39.435 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 118 == 0 2023-01-25T06:33:39.436 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T06:33:39.436 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T06:33:39.437 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T06:33:39.437 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T06:33:39.437 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T06:33:39.437 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T06:33:39.438 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T06:33:39.911 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=118 2023-01-25T06:33:39.911 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T06:33:39.912 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:33:39.912 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:33:40.522 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 118 = 118 2023-01-25T06:33:40.523 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T06:33:40.523 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T06:33:40.523 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:154: TEST_balancer2: OK=no 2023-01-25T06:33:40.524 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: seq 1 25 2023-01-25T06:33:40.524 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:33:40.525 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:33:45.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:33:45.540 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:33:45.540 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:33:50.542 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:33:50.556 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:33:50.557 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:33:55.558 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:33:55.572 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:33:55.572 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:00.574 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:00.587 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:00.587 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:05.589 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:05.602 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:05.603 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:10.605 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:10.618 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:10.618 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:15.620 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:15.632 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:15.632 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:20.634 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:20.648 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:20.648 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:25.650 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:25.662 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:25.663 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:30.665 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:30.678 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:155: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:30.679 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:157: TEST_balancer2: sleep 5 2023-01-25T06:34:35.680 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:158: TEST_balancer2: grep -q 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:34:35.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:160: TEST_balancer2: OK=yes 2023-01-25T06:34:35.694 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:161: TEST_balancer2: break 2023-01-25T06:34:35.695 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:164: TEST_balancer2: test yes = yes 2023-01-25T06:34:35.695 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:166: TEST_balancer2: sleep 10 2023-01-25T06:34:45.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:167: TEST_balancer2: wait_for_clean 2023-01-25T06:34:45.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T06:34:45.697 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T06:34:45.698 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T06:34:45.698 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T06:34:45.698 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T06:34:45.699 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T06:34:45.699 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T06:34:45.700 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T06:34:45.700 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T06:34:45.700 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T06:34:45.924 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T06:34:45.924 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T06:34:45.925 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T06:34:45.925 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T06:34:45.925 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T06:34:46.369 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T06:34:46.369 INFO:tasks.workunit.client.0.smithi040.stderr:1 2023-01-25T06:34:46.369 INFO:tasks.workunit.client.0.smithi040.stderr:2 2023-01-25T06:34:46.370 INFO:tasks.workunit.client.0.smithi040.stderr:3 2023-01-25T06:34:46.370 INFO:tasks.workunit.client.0.smithi040.stderr:4' 2023-01-25T06:34:46.370 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T06:34:46.370 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:34:46.371 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T06:34:46.555 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836534 2023-01-25T06:34:46.555 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836534 2023-01-25T06:34:46.556 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836534' 2023-01-25T06:34:46.556 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:34:46.556 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T06:34:46.719 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673011 2023-01-25T06:34:46.719 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673011 2023-01-25T06:34:46.719 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836534 1-42949673011' 2023-01-25T06:34:46.720 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:34:46.720 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T06:34:46.870 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509481 2023-01-25T06:34:46.870 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509481 2023-01-25T06:34:46.870 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836534 1-42949673011 2-64424509481' 2023-01-25T06:34:46.871 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:34:46.871 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T06:34:47.028 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345953 2023-01-25T06:34:47.028 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345953 2023-01-25T06:34:47.028 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836534 1-42949673011 2-64424509481 3-85899345953' 2023-01-25T06:34:47.029 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:34:47.029 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-25T06:34:47.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182422 2023-01-25T06:34:47.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182422 2023-01-25T06:34:47.182 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836534 1-42949673011 2-64424509481 3-85899345953 4-107374182422' 2023-01-25T06:34:47.183 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:34:47.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836534 2023-01-25T06:34:47.183 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:34:47.185 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T06:34:47.186 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836534 2023-01-25T06:34:47.187 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:34:47.188 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836534 2023-01-25T06:34:47.188 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2023-01-25T06:34:47.188 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.0 seq 21474836534 2023-01-25T06:34:47.189 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:34:47.642 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836534 -lt 21474836534 2023-01-25T06:34:47.642 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:34:47.643 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673011 2023-01-25T06:34:47.643 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:34:47.644 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T06:34:47.645 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:34:47.646 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673011 2023-01-25T06:34:47.646 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673011 2023-01-25T06:34:47.647 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.1 seq 42949673011 2023-01-25T06:34:47.647 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2023-01-25T06:34:47.648 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T06:34:48.110 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673009 -lt 42949673011 2023-01-25T06:34:48.110 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: sleep 1 2023-01-25T06:34:49.112 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: '[' 300 -eq 0 ']' 2023-01-25T06:34:49.113 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T06:34:49.545 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673011 -lt 42949673011 2023-01-25T06:34:49.546 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:34:49.546 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509481 2023-01-25T06:34:49.547 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:34:49.548 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T06:34:49.550 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509481 2023-01-25T06:34:49.550 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:34:49.551 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509481 2023-01-25T06:34:49.551 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2023-01-25T06:34:49.552 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.2 seq 64424509481 2023-01-25T06:34:49.552 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T06:34:49.970 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509481 -lt 64424509481 2023-01-25T06:34:49.970 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:34:49.971 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345953 2023-01-25T06:34:49.971 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:34:49.972 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T06:34:49.972 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:34:49.973 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345953 2023-01-25T06:34:49.974 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345953 2023-01-25T06:34:49.974 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2023-01-25T06:34:49.974 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.3 seq 85899345953 2023-01-25T06:34:49.975 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T06:34:50.450 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345953 -lt 85899345953 2023-01-25T06:34:50.451 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:34:50.451 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182422 2023-01-25T06:34:50.451 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:34:50.452 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-25T06:34:50.453 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182422 2023-01-25T06:34:50.454 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:34:50.455 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182422 2023-01-25T06:34:50.456 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182422' 2023-01-25T06:34:50.456 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.4 seq 107374182422 2023-01-25T06:34:50.457 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-25T06:34:50.916 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182423 -lt 107374182422 2023-01-25T06:34:50.917 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T06:34:50.918 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:34:50.918 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:34:51.467 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 118 == 0 2023-01-25T06:34:51.467 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T06:34:51.468 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T06:34:51.468 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T06:34:51.468 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T06:34:51.469 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T06:34:51.469 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T06:34:51.469 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T06:34:51.948 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=118 2023-01-25T06:34:51.949 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T06:34:51.949 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:34:51.950 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:34:52.526 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 118 = 118 2023-01-25T06:34:52.526 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T06:34:52.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T06:34:52.527 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:168: TEST_balancer2: ceph osd df 2023-01-25T06:34:52.990 INFO:tasks.workunit.client.0.smithi040.stdout:ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2023-01-25T06:34:52.991 INFO:tasks.workunit.client.0.smithi040.stdout: 0 hdd 0.09769 1.00000 100 GiB 20 MiB 296 KiB 0 B 20 MiB 100 GiB 0.02 1.00 71 up 2023-01-25T06:34:52.991 INFO:tasks.workunit.client.0.smithi040.stdout: 1 hdd 0.09769 1.00000 100 GiB 20 MiB 296 KiB 0 B 20 MiB 100 GiB 0.02 1.00 71 up 2023-01-25T06:34:52.991 INFO:tasks.workunit.client.0.smithi040.stdout: 2 hdd 0.09769 1.00000 100 GiB 20 MiB 296 KiB 0 B 20 MiB 100 GiB 0.02 1.00 70 up 2023-01-25T06:34:52.991 INFO:tasks.workunit.client.0.smithi040.stdout: 3 hdd 0.09769 1.00000 100 GiB 20 MiB 296 KiB 0 B 20 MiB 100 GiB 0.02 1.00 71 up 2023-01-25T06:34:52.992 INFO:tasks.workunit.client.0.smithi040.stdout: 4 hdd 0.09769 1.00000 100 GiB 20 MiB 296 KiB 0 B 20 MiB 100 GiB 0.02 1.00 71 up 2023-01-25T06:34:52.992 INFO:tasks.workunit.client.0.smithi040.stdout: TOTAL 500 GiB 101 MiB 1.4 MiB 0 B 99 MiB 500 GiB 0.02 2023-01-25T06:34:52.992 INFO:tasks.workunit.client.0.smithi040.stdout:MIN/MAX VAR: 1.00/1.00 STDDEV: 0 2023-01-25T06:34:53.003 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:170: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:34:53.003 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:170: TEST_balancer2: jq '.nodes[0].pgs' 2023-01-25T06:34:53.388 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:170: TEST_balancer2: PGS=71 2023-01-25T06:34:53.389 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:171: TEST_balancer2: test 71 -ge 70 2023-01-25T06:34:53.389 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:172: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:34:53.390 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:172: TEST_balancer2: jq '.nodes[1].pgs' 2023-01-25T06:34:53.793 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:172: TEST_balancer2: PGS=71 2023-01-25T06:34:53.793 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:173: TEST_balancer2: test 71 -ge 70 2023-01-25T06:34:53.794 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:174: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:34:53.794 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:174: TEST_balancer2: jq '.nodes[2].pgs' 2023-01-25T06:34:54.209 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:174: TEST_balancer2: PGS=70 2023-01-25T06:34:54.209 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:175: TEST_balancer2: test 70 -ge 70 2023-01-25T06:34:54.210 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:176: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:34:54.210 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:176: TEST_balancer2: jq '.nodes[3].pgs' 2023-01-25T06:34:54.691 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:176: TEST_balancer2: PGS=71 2023-01-25T06:34:54.691 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:177: TEST_balancer2: test 71 -ge 70 2023-01-25T06:34:54.692 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:178: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:34:54.692 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:178: TEST_balancer2: jq '.nodes[4].pgs' 2023-01-25T06:34:55.092 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:178: TEST_balancer2: PGS=71 2023-01-25T06:34:55.092 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:179: TEST_balancer2: test 71 -ge 70 2023-01-25T06:34:55.092 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:181: TEST_balancer2: create_pool test2 132 2023-01-25T06:34:55.093 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 132 2023-01-25T06:34:55.722 INFO:tasks.workunit.client.0.smithi040.stderr:pool 'test2' already exists 2023-01-25T06:34:55.737 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-01-25T06:34:56.739 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:184: TEST_balancer2: OK=no 2023-01-25T06:34:56.739 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: seq 1 25 2023-01-25T06:34:56.741 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:34:56.741 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:01.745 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:01.746 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:01.761 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=1 2023-01-25T06:35:01.761 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 1 = 2 2023-01-25T06:35:01.762 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:35:01.762 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:06.763 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:06.763 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:06.779 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=1 2023-01-25T06:35:06.779 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 1 = 2 2023-01-25T06:35:06.779 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:35:06.780 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:11.782 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:11.783 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:11.799 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=1 2023-01-25T06:35:11.799 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 1 = 2 2023-01-25T06:35:11.799 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:35:11.800 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:16.802 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:16.802 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:16.817 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=1 2023-01-25T06:35:16.817 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 1 = 2 2023-01-25T06:35:16.817 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:35:16.817 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:21.820 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:21.821 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:21.836 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=1 2023-01-25T06:35:21.836 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 1 = 2 2023-01-25T06:35:21.836 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:185: TEST_balancer2: for i in $(seq 1 25) 2023-01-25T06:35:21.836 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:187: TEST_balancer2: sleep 5 2023-01-25T06:35:26.839 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: grep 'Optimization plan is almost perfect' td/balancer/mgr.x.log 2023-01-25T06:35:26.840 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: wc -l 2023-01-25T06:35:26.856 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:188: TEST_balancer2: COUNT=2 2023-01-25T06:35:26.856 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:189: TEST_balancer2: test 2 = 2 2023-01-25T06:35:26.856 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:191: TEST_balancer2: OK=yes 2023-01-25T06:35:26.856 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:192: TEST_balancer2: break 2023-01-25T06:35:26.857 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:195: TEST_balancer2: test yes = yes 2023-01-25T06:35:26.857 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:197: TEST_balancer2: sleep 10 2023-01-25T06:35:36.859 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:198: TEST_balancer2: wait_for_clean 2023-01-25T06:35:36.859 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: local cmd= 2023-01-25T06:35:36.859 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: local num_active_clean=-1 2023-01-25T06:35:36.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cur_active_clean 2023-01-25T06:35:36.860 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-01-25T06:35:36.860 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: get_timeout_delays 90 .1 2023-01-25T06:35:36.861 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: shopt -q -o xtrace 2023-01-25T06:35:36.861 INFO:tasks.workunit.client.0.smithi040.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: echo true 2023-01-25T06:35:36.861 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: get_timeout_delays: local trace=true 2023-01-25T06:35:36.861 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: true 2023-01-25T06:35:36.862 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: get_timeout_delays: shopt -u -o xtrace 2023-01-25T06:35:37.088 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local -a delays 2023-01-25T06:35:37.088 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local -i loop=0 2023-01-25T06:35:37.088 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: flush_pg_stats 2023-01-25T06:35:37.088 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: flush_pg_stats: local timeout=300 2023-01-25T06:35:37.089 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ceph osd ls 2023-01-25T06:35:37.559 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: ids='0 2023-01-25T06:35:37.559 INFO:tasks.workunit.client.0.smithi040.stderr:1 2023-01-25T06:35:37.560 INFO:tasks.workunit.client.0.smithi040.stderr:2 2023-01-25T06:35:37.560 INFO:tasks.workunit.client.0.smithi040.stderr:3 2023-01-25T06:35:37.560 INFO:tasks.workunit.client.0.smithi040.stderr:4' 2023-01-25T06:35:37.560 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2130: flush_pg_stats: seqs= 2023-01-25T06:35:37.560 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:35:37.561 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-01-25T06:35:37.721 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=21474836547 2023-01-25T06:35:37.722 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 21474836547 2023-01-25T06:35:37.722 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836547' 2023-01-25T06:35:37.722 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:35:37.722 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-01-25T06:35:37.886 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=42949673023 2023-01-25T06:35:37.886 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 42949673023 2023-01-25T06:35:37.887 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836547 1-42949673023' 2023-01-25T06:35:37.887 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:35:37.887 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-01-25T06:35:38.042 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=64424509493 2023-01-25T06:35:38.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 64424509493 2023-01-25T06:35:38.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493' 2023-01-25T06:35:38.043 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:35:38.044 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-01-25T06:35:38.198 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=85899345965 2023-01-25T06:35:38.198 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 85899345965 2023-01-25T06:35:38.198 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345965' 2023-01-25T06:35:38.199 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: for osd in $ids 2023-01-25T06:35:38.199 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-01-25T06:35:38.362 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seq=107374182434 2023-01-25T06:35:38.362 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: test -z 107374182434 2023-01-25T06:35:38.363 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345965 4-107374182434' 2023-01-25T06:35:38.363 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:35:38.363 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 0-21474836547 2023-01-25T06:35:38.364 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:35:38.365 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=0 2023-01-25T06:35:38.366 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 0-21474836547 2023-01-25T06:35:38.366 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:35:38.368 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=21474836547 2023-01-25T06:35:38.368 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2023-01-25T06:35:38.368 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.0 seq 21474836547 2023-01-25T06:35:38.369 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 0 2023-01-25T06:35:38.829 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 21474836547 -lt 21474836547 2023-01-25T06:35:38.829 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:35:38.830 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 1-42949673023 2023-01-25T06:35:38.830 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:35:38.832 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=1 2023-01-25T06:35:38.833 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 1-42949673023 2023-01-25T06:35:38.833 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:35:38.835 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=42949673023 2023-01-25T06:35:38.835 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2023-01-25T06:35:38.835 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.1 seq 42949673023 2023-01-25T06:35:38.836 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 1 2023-01-25T06:35:39.321 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 42949673023 -lt 42949673023 2023-01-25T06:35:39.321 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:35:39.322 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 2-64424509493 2023-01-25T06:35:39.322 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:35:39.324 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=2 2023-01-25T06:35:39.325 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 2-64424509493 2023-01-25T06:35:39.326 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:35:39.327 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=64424509493 2023-01-25T06:35:39.328 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2023-01-25T06:35:39.328 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.2 seq 64424509493 2023-01-25T06:35:39.328 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 2 2023-01-25T06:35:39.800 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 64424509493 -lt 64424509493 2023-01-25T06:35:39.800 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:35:39.801 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 3-85899345965 2023-01-25T06:35:39.801 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:35:39.803 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=3 2023-01-25T06:35:39.804 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 3-85899345965 2023-01-25T06:35:39.804 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:35:39.806 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=85899345965 2023-01-25T06:35:39.806 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.3 seq 85899345965 2023-01-25T06:35:39.807 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2023-01-25T06:35:39.807 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 3 2023-01-25T06:35:40.261 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 85899345965 -lt 85899345965 2023-01-25T06:35:40.261 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: flush_pg_stats: for s in $seqs 2023-01-25T06:35:40.262 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: echo 4-107374182434 2023-01-25T06:35:40.262 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: cut -d - -f 1 2023-01-25T06:35:40.263 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: flush_pg_stats: osd=4 2023-01-25T06:35:40.264 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: echo 4-107374182434 2023-01-25T06:35:40.265 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: cut -d - -f 2 2023-01-25T06:35:40.266 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: seq=107374182434 2023-01-25T06:35:40.267 INFO:tasks.workunit.client.0.smithi040.stdout:waiting osd.4 seq 107374182434 2023-01-25T06:35:40.267 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2023-01-25T06:35:40.268 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: ceph osd last-stat-seq 4 2023-01-25T06:35:40.690 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: test 107374182434 -lt 107374182434 2023-01-25T06:35:40.690 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: get_num_pgs 2023-01-25T06:35:40.691 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:35:40.691 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:35:41.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: test 250 == 0 2023-01-25T06:35:41.222 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: true 2023-01-25T06:35:41.223 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: get_num_active_clean 2023-01-25T06:35:41.223 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_active_clean: local expression 2023-01-25T06:35:41.223 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-01-25T06:35:41.224 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-01-25T06:35:41.224 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: ceph --format json pg dump pgs 2023-01-25T06:35:41.224 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-01-25T06:35:41.721 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: cur_active_clean=250 2023-01-25T06:35:41.722 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: get_num_pgs 2023-01-25T06:35:41.723 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: ceph --format json status 2023-01-25T06:35:41.723 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: get_num_pgs: jq .pgmap.num_pgs 2023-01-25T06:35:42.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: test 250 = 250 2023-01-25T06:35:42.268 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: break 2023-01-25T06:35:42.269 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 0 2023-01-25T06:35:42.269 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:199: TEST_balancer2: ceph osd df 2023-01-25T06:35:42.699 INFO:tasks.workunit.client.0.smithi040.stdout:ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2023-01-25T06:35:42.699 INFO:tasks.workunit.client.0.smithi040.stdout: 0 hdd 0.09769 1.00000 100 GiB 21 MiB 440 KiB 0 B 21 MiB 100 GiB 0.02 1.00 150 up 2023-01-25T06:35:42.700 INFO:tasks.workunit.client.0.smithi040.stdout: 1 hdd 0.09769 1.00000 100 GiB 21 MiB 440 KiB 0 B 21 MiB 100 GiB 0.02 1.00 151 up 2023-01-25T06:35:42.700 INFO:tasks.workunit.client.0.smithi040.stdout: 2 hdd 0.09769 1.00000 100 GiB 21 MiB 440 KiB 0 B 21 MiB 100 GiB 0.02 1.00 149 up 2023-01-25T06:35:42.700 INFO:tasks.workunit.client.0.smithi040.stdout: 3 hdd 0.09769 1.00000 100 GiB 21 MiB 440 KiB 0 B 21 MiB 100 GiB 0.02 1.00 150 up 2023-01-25T06:35:42.700 INFO:tasks.workunit.client.0.smithi040.stdout: 4 hdd 0.09769 1.00000 100 GiB 21 MiB 440 KiB 0 B 20 MiB 100 GiB 0.02 1.00 150 up 2023-01-25T06:35:42.700 INFO:tasks.workunit.client.0.smithi040.stdout: TOTAL 500 GiB 105 MiB 2.1 MiB 0 B 103 MiB 500 GiB 0.02 2023-01-25T06:35:42.701 INFO:tasks.workunit.client.0.smithi040.stdout:MIN/MAX VAR: 1.00/1.00 STDDEV: 0 2023-01-25T06:35:42.713 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:203: TEST_balancer2: expr 150 - 2 2023-01-25T06:35:42.714 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:203: TEST_balancer2: MIN=148 2023-01-25T06:35:42.715 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:204: TEST_balancer2: expr 150 + 2 2023-01-25T06:35:42.716 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:204: TEST_balancer2: MAX=152 2023-01-25T06:35:42.718 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:205: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:35:42.718 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:205: TEST_balancer2: jq '.nodes[0].pgs' 2023-01-25T06:35:43.131 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:205: TEST_balancer2: PGS=150 2023-01-25T06:35:43.131 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:206: TEST_balancer2: test 150 -ge 148 -a 150 -le 152 2023-01-25T06:35:43.132 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:207: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:35:43.132 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:207: TEST_balancer2: jq '.nodes[1].pgs' 2023-01-25T06:35:43.540 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:207: TEST_balancer2: PGS=151 2023-01-25T06:35:43.541 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:208: TEST_balancer2: test 151 -ge 148 -a 151 -le 152 2023-01-25T06:35:43.542 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:209: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:35:43.542 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:209: TEST_balancer2: jq '.nodes[2].pgs' 2023-01-25T06:35:43.948 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:209: TEST_balancer2: PGS=149 2023-01-25T06:35:43.949 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:210: TEST_balancer2: test 149 -ge 148 -a 149 -le 152 2023-01-25T06:35:43.950 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:211: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:35:43.950 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:211: TEST_balancer2: jq '.nodes[3].pgs' 2023-01-25T06:35:44.356 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:211: TEST_balancer2: PGS=150 2023-01-25T06:35:44.357 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:212: TEST_balancer2: test 150 -ge 148 -a 150 -le 152 2023-01-25T06:35:44.357 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:213: TEST_balancer2: ceph osd df --format=json-pretty 2023-01-25T06:35:44.358 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:213: TEST_balancer2: jq '.nodes[4].pgs' 2023-01-25T06:35:44.752 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:213: TEST_balancer2: PGS=150 2023-01-25T06:35:44.753 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:214: TEST_balancer2: test 150 -ge 148 -a 150 -le 152 2023-01-25T06:35:44.753 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh:216: TEST_balancer2: teardown td/balancer 2023-01-25T06:35:44.753 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/balancer 2023-01-25T06:35:44.753 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-01-25T06:35:44.754 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/balancer KILL 2023-01-25T06:35:44.754 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T06:35:44.755 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T06:35:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T06:35:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T06:35:44.755 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T06:35:44.870 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T06:35:44.871 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T06:35:44.872 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T06:35:44.872 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T06:35:44.875 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T06:35:44.875 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T06:35:44.875 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T06:35:44.877 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:35:44.877 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T06:35:44.878 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T06:35:44.878 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:35:44.879 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:35:44.882 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-01-25T06:35:44.882 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/balancer 2023-01-25T06:35:44.920 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T06:35:44.921 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:35:44.921 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:35:44.921 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22991 2023-01-25T06:35:44.923 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T06:35:44.923 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T06:35:44.924 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: code=0 2023-01-25T06:35:44.924 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: main: teardown td/balancer 0 2023-01-25T06:35:44.924 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/balancer 2023-01-25T06:35:44.925 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-01-25T06:35:44.925 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/balancer KILL 2023-01-25T06:35:44.925 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-01-25T06:35:44.925 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-01-25T06:35:44.926 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-01-25T06:35:44.926 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-01-25T06:35:44.926 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-01-25T06:35:44.928 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-01-25T06:35:44.929 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-01-25T06:35:44.931 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-01-25T06:35:44.931 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-01-25T06:35:44.933 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-01-25T06:35:44.934 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-01-25T06:35:44.934 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-01-25T06:35:44.936 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:35:44.936 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-01-25T06:35:44.937 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-01-25T06:35:44.937 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T06:35:44.938 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:35:44.940 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-01-25T06:35:44.940 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/balancer 2023-01-25T06:35:44.941 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-01-25T06:35:44.942 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-01-25T06:35:44.942 INFO:tasks.workunit.client.0.smithi040.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.22991 2023-01-25T06:35:44.942 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.22991 2023-01-25T06:35:44.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-01-25T06:35:44.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-01-25T06:35:44.943 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: main: return 0 2023-01-25T06:35:44.945 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-25T06:35:44.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-25T06:35:44.957 INFO:tasks.workunit:Stopping ['mgr'] on client.0... 2023-01-25T06:35:44.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-01-25T06:35:45.247 DEBUG:teuthology.parallel:result is None 2023-01-25T06:35:45.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T06:35:45.258 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T06:35:45.259 DEBUG:teuthology.orchestra.run.smithi040:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T06:35:45.314 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T06:35:45.314 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-01-25T06:35:45.328 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-01-25T06:35:45.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-01-25T06:35:45.372 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-01-25T06:35:45.381 INFO:teuthology.task.clock:Checking final clock skew... 2023-01-25T06:35:45.381 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T06:35:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2023-01-25T06:35:45.406 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2023-01-25T06:35:45.406 INFO:teuthology.orchestra.run.smithi040.stdout:+hv01.front.sepi 216.240.36.24 3 u 46 64 377 0.074 -3.309 0.951 2023-01-25T06:35:45.406 INFO:teuthology.orchestra.run.smithi040.stdout:+hv02.front.sepi 152.44.36.235 3 u 44 64 377 0.069 1.827 0.938 2023-01-25T06:35:45.407 INFO:teuthology.orchestra.run.smithi040.stdout:*hv03.front.sepi 206.82.28.3 3 u 47 64 377 0.097 1.471 1.195 2023-01-25T06:35:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 0.000 0.000 2023-01-25T06:35:45.408 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-25T06:35:45.417 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-25T06:35:45.417 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-25T06:35:45.426 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-25T06:35:45.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-25T06:35:45.443 INFO:teuthology.task.internal:Duration was 924.660422 seconds 2023-01-25T06:35:45.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-25T06:35:45.453 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-25T06:35:45.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-25T06:35:45.501 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-25T06:35:45.501 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-01-25T06:35:45.501 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-25T06:35:45.513 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-25T06:35:45.514 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-25T06:35:45.602 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-25T06:35:45.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-25T06:35:45.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-25T06:35:45.746 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-25T06:35:45.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-25T06:35:45.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-25T06:35:45.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:35:45.819 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-01-25T06:35:45.836 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-25T06:35:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T06:35:45.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-25T06:35:45.852 INFO:teuthology.task.internal:Transferring archived files... 2023-01-25T06:35:45.854 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136911/remote/smithi040 2023-01-25T06:35:45.858 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-25T06:35:45.895 INFO:teuthology.task.internal:Removing archive directory... 2023-01-25T06:35:45.896 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-25T06:35:45.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-25T06:35:45.950 INFO:teuthology.task.internal:Not uploading archives. 2023-01-25T06:35:45.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-25T06:35:45.960 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-25T06:35:45.961 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-25T06:35:45.968 INFO:teuthology.orchestra.run.smithi040.stdout: 924115 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 25 06:35 /home/ubuntu/cephtest 2023-01-25T06:35:45.971 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-25T06:35:45.992 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} duration: 924.6604223251343 flavor: default owner: scheduled_yuriw@teuthology success: true 2023-01-25T06:35:45.993 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T06:35:46.068 INFO:teuthology.run:pass