2024-05-01T12:44:21.656 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-05-01T12:44:21.656 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw:notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/others/{0-install supported-distros/{centos_latest} test_others}} --archive /home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921 --owner scheduled_alimasa@teuthology --verbose --name alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi -- /home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921/orig.config.yaml 2024-05-01T12:44:21.690 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-01T12:44:21.774 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921 branch: wip-alimasa-bn-test description: rgw:notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/others/{0-install supported-distros/{centos_latest} test_others}} email: null first_in_suite: false job_id: '7682921' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-alimasa-bn-test ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= setgroup: ceph setuser: ceph global: osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: abc5b10088b379036a15386486ef8fee07d8cc3c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: abc5b10088b379036a15386486ef8fee07d8cc3c rgw: frontend: beast realm: MyRealm storage classes: LUKEWARM, FROZEN zone: MyZone zonegroup: MyZoneGroup selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-alimasa-bn-test-testing sha1: 971a59c7a133e90e9bfb891a1b11587cefa2ea03 owner: scheduled_alimasa@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: abc5b10088b379036a15386486ef8fee07d8cc3c sleep_before_teardown: 0 suite: rgw:notifications suite_branch: wip-alimasa-bn-test-testing suite_path: /home/teuthworker/src/github.com_AliMasarweh_ceph_971a59c7a133e90e9bfb891a1b11587cefa2ea03/qa suite_relpath: qa suite_repo: https://github.com/AliMasarweh/ceph.git suite_sha1: 971a59c7a133e90e9bfb891a1b11587cefa2ea03 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlyqt9Hkbw7aMOo6S+f2Sg5D459tcj8ks9Myzxf7bJEJAVb0kTxRr/H1C7BT/8wL/anPuInHWl0ssxj2FpX9Y= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLG4sVglRCzJsU4+TxHf4i78tWMLMAmZKIVu1h9DesveLneCAUiVtS/k9oSLU85IJbAP8zr+7tG7CDeebi09xQ= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null client.1: null - notification-tests: client.0: rgw_server: client.0 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-05-01_12:37:01 tube: smithi user: alimasa verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-01T12:44:21.774 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_AliMasarweh_ceph_971a59c7a133e90e9bfb891a1b11587cefa2ea03/qa; will attempt to use it 2024-05-01T12:44:21.775 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_AliMasarweh_ceph_971a59c7a133e90e9bfb891a1b11587cefa2ea03/qa/tasks 2024-05-01T12:44:21.775 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-01T12:44:21.777 INFO:teuthology.task.internal:Checking packages... 2024-05-01T12:44:21.799 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'abc5b10088b379036a15386486ef8fee07d8cc3c' 2024-05-01T12:44:21.800 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-01T12:44:21.800 INFO:teuthology.packaging:ref: None 2024-05-01T12:44:21.800 INFO:teuthology.packaging:tag: None 2024-05-01T12:44:21.800 INFO:teuthology.packaging:branch: wip-alimasa-bn-test 2024-05-01T12:44:21.800 INFO:teuthology.packaging:sha1: abc5b10088b379036a15386486ef8fee07d8cc3c 2024-05-01T12:44:21.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-alimasa-bn-test 2024-05-01T12:44:22.024 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3122.gabc5b100 2024-05-01T12:44:22.025 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-01T12:44:22.032 INFO:teuthology.task.internal:no buildpackages task found 2024-05-01T12:44:22.032 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-01T12:44:22.055 INFO:teuthology.task.internal:Saving configuration 2024-05-01T12:44:22.067 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-01T12:44:22.074 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-01T12:44:22.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-01 12:39:12.857888', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlyqt9Hkbw7aMOo6S+f2Sg5D459tcj8ks9Myzxf7bJEJAVb0kTxRr/H1C7BT/8wL/anPuInHWl0ssxj2FpX9Y='} 2024-05-01T12:44:22.121 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-01 12:39:12.856506', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLG4sVglRCzJsU4+TxHf4i78tWMLMAmZKIVu1h9DesveLneCAUiVtS/k9oSLU85IJbAP8zr+7tG7CDeebi09xQ='} 2024-05-01T12:44:22.121 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-01T12:44:22.128 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-01T12:44:22.128 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-01T12:44:22.128 INFO:teuthology.run_tasks:Running task console_log... 2024-05-01T12:44:22.197 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc5844ef70>, signals=[15]) 2024-05-01T12:44:22.197 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-01T12:44:22.204 INFO:teuthology.task.internal:Opening connections... 2024-05-01T12:44:22.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:44:22.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-01T12:44:22.283 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:44:22.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-01T12:44:22.355 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-01T12:44:22.362 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-05-01T12:44:22.380 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-05-01T12:44:22.381 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-05-01T12:44:22.438 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-05-01T12:44:22.438 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-05-01T12:44:22.438 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-05-01T12:44:22.438 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-05-01T12:44:22.438 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-01T12:44:22.439 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-01T12:44:22.440 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-05-01T12:44:22.459 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-05-01T12:44:22.478 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-05-01T12:44:22.478 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-05-01T12:44:22.536 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-01T12:44:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-01T12:44:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-01T12:44:22.537 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-05-01T12:44:22.559 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-01T12:44:22.567 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-01T12:44:22.573 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-01T12:44:22.573 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-05-01T12:44:22.575 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-05-01T12:44:22.592 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-01T12:44:22.600 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-01T12:44:22.600 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-05-01T12:44:22.633 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-05-01T12:44:22.656 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-01T12:44:22.711 INFO:teuthology.run_tasks:Running task kernel... 2024-05-01T12:44:22.726 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-05-01T12:44:22.726 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-05-01T12:44:22.726 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-01T12:44:22.727 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-01T12:44:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-01T12:44:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-01T12:44:22.744 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-05-01T12:44:22.744 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-05-01T12:44:22.802 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-437.el9.x86_64 2024-05-01T12:44:22.802 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-437.el9.x86_64 2024-05-01T12:44:22.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-05-01T12:44:22.803 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-437.el9.x86_64 2024-05-01T12:44:22.803 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-437.el9.x86_64 2024-05-01T12:44:22.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-05-01T12:44:24.893 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-05-01T12:44:26.550 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-05-01T12:44:26.588 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-05-01T12:44:29.808 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 19 MB 00:02 2024-05-01T12:44:31.476 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.2 MB 00:01 2024-05-01T12:44:32.893 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 16 kB 00:00 2024-05-01T12:44:35.373 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-01T12:44:36.412 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-05-01T12:44:36.715 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 16 kB 00:00 2024-05-01T12:44:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 21 MB 00:03 2024-05-01T12:44:41.645 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-01T12:44:43.601 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-01T12:44:43.601 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-01T12:44:43.696 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:44:43.697 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:44:43.697 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:44:43.804 DEBUG:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2024-05-01T12:44:44.323 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Wed 01 May 2024 12:44:41 PM UTC. 2024-05-01T12:44:44.451 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 255 k 2024-05-01T12:44:44.464 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-05-01T12:44:44.465 INFO:teuthology.orchestra.run.smithi050.stderr:Operation aborted. 2024-05-01T12:44:44.522 INFO:teuthology.orchestra.run.smithi050.stdout:Is this ok [y/N]: 2024-05-01T12:44:44.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2024-05-01T12:44:45.038 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:04 ago on Wed 01 May 2024 12:44:41 PM UTC. 2024-05-01T12:44:45.083 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-01T12:44:45.159 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:44:45.159 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 255 k 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-05-01T12:44:45.160 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:44:45.588 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.5 MB/s | 255 kB 00:00 2024-05-01T12:44:45.588 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:44:45.589 INFO:teuthology.orchestra.run.smithi050.stdout:Total 595 kB/s | 255 kB 00:00 2024-05-01T12:44:45.610 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:44:45.807 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:44:45.807 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:44:45.812 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:44:45.812 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:44:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:44:46.048 INFO:teuthology.orchestra.run.smithi050.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-01T12:44:46.048 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:46.705 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:46.705 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-01T12:44:47.139 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-01T12:44:47.139 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-01T12:44:47.235 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:44:47.236 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:44:47.236 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:44:47.337 DEBUG:teuthology.orchestra.run.smithi063:> echo no | sudo yum reinstall kernel || true 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalled: 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:44:47.543 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-05-01T12:44:47.591 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-01T12:44:47.592 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-01T12:44:47.592 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-01T12:44:47.592 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-01T12:44:47.592 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-01T12:44:47.623 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-05-01T12:44:47.645 DEBUG:teuthology.parallel:result is None 2024-05-01T12:44:47.853 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:02 ago on Wed 01 May 2024 12:44:45 PM UTC. 2024-05-01T12:44:47.976 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:44:47.985 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:47.985 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-05-01T12:44:47.985 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:47.985 INFO:teuthology.orchestra.run.smithi063.stdout:Reinstalling: 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 255 k 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 0 2024-05-01T12:44:47.986 INFO:teuthology.orchestra.run.smithi063.stderr:Operation aborted. 2024-05-01T12:44:48.048 INFO:teuthology.orchestra.run.smithi063.stdout:Is this ok [y/N]: 2024-05-01T12:44:48.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum reinstall -y kernel || true 2024-05-01T12:44:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:03 ago on Wed 01 May 2024 12:44:45 PM UTC. 2024-05-01T12:44:48.683 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout:Reinstalling: 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:44:48.684 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:48.685 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 255 k 2024-05-01T12:44:48.685 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 0 2024-05-01T12:44:48.685 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:44:49.118 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-05-01T12:44:49.119 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:44:49.119 INFO:teuthology.orchestra.run.smithi063.stdout:Total 588 kB/s | 255 kB 00:00 2024-05-01T12:44:49.134 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:44:49.314 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:44:49.314 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:44:49.319 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:44:49.319 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:44:49.450 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:44:49.528 INFO:teuthology.orchestra.run.smithi063.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-01T12:44:49.528 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout:Reinstalled: 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:44:50.991 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:44:51.176 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-05-01T12:44:51.224 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-01T12:44:51.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-01T12:44:51.224 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-01T12:44:51.224 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-01T12:44:51.224 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-01T12:44:51.259 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-05-01T12:44:51.288 DEBUG:teuthology.parallel:result is None 2024-05-01T12:44:51.289 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-01T12:44:51.298 INFO:teuthology.task.internal:Creating test directory... 2024-05-01T12:44:51.298 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-01T12:44:51.301 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-01T12:44:51.344 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-01T12:44:51.351 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-01T12:44:51.365 INFO:teuthology.task.internal:Creating archive directory... 2024-05-01T12:44:51.366 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-01T12:44:51.368 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-01T12:44:51.414 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-01T12:44:51.421 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-01T12:44:51.421 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-01T12:44:51.434 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-01T12:44:51.469 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-01T12:44:51.480 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-01T12:44:51.489 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-01T12:44:51.498 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-01T12:44:51.500 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-01T12:44:51.507 INFO:teuthology.task.internal:Configuring sudo... 2024-05-01T12:44:51.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-01T12:44:51.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-01T12:44:51.564 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-01T12:44:51.574 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-01T12:44:51.574 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-01T12:44:51.594 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-01T12:44:51.619 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-01T12:44:51.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-01T12:44:51.749 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-01T12:44:51.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-01T12:44:51.879 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:44:51.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-01T12:44:51.948 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-01T12:44:51.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-01T12:44:52.042 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-01T12:44:52.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-01T12:44:52.163 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:44:52.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-01T12:44:52.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-05-01T12:44:52.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-05-01T12:44:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-01T12:44:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-01T12:44:52.790 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-01T12:44:52.798 INFO:teuthology.task.internal:Starting timer... 2024-05-01T12:44:52.798 INFO:teuthology.run_tasks:Running task pcp... 2024-05-01T12:44:52.808 INFO:teuthology.run_tasks:Running task selinux... 2024-05-01T12:44:52.817 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-01T12:44:52.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-05-01T12:44:52.892 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-05-01T12:44:52.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-05-01T12:44:52.932 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-05-01T12:44:52.934 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-01T12:44:52.935 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-05-01T12:44:52.959 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-05-01T12:44:52.959 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-05-01T12:44:52.998 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-05-01T12:44:52.998 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi063.front.sepia.ceph.com': 'permissive'} 2024-05-01T12:44:52.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-01T12:44:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-01T12:44:53.027 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-01T12:44:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-01T12:44:53.063 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-01T12:44:53.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-05-01T12:44:53.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-05-01T12:44:53.134 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-01T12:44:53.144 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-01T12:44:53.161 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-01T12:44:53.162 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-01T12:48:49.302 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com')] 2024-05-01T12:48:49.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-05-01T12:48:49.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-01T12:48:49.383 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-05-01T12:48:49.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-05-01T12:48:49.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-05-01T12:48:49.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-01T12:48:49.535 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-05-01T12:48:49.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-05-01T12:48:49.617 INFO:teuthology.run_tasks:Running task clock... 2024-05-01T12:48:49.628 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-01T12:48:49.628 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-01T12:48:49.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-01T12:48:49.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-01T12:48:49.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-01T12:48:49.658 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-01T12:48:49.672 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-01T12:48:49.686 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-01T12:48:49.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-05-01T12:48:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-01T12:48:49.707 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-05-01T12:48:49.721 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-01T12:48:49.723 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-05-01T12:48:49.734 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-05-01T12:48:49.734 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-01T12:48:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-01T12:48:49.761 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-01T12:48:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.800 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.825 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-01T12:48:49.830 INFO:teuthology.run_tasks:Running task install... 2024-05-01T12:48:49.838 DEBUG:teuthology.task.install:project ceph 2024-05-01T12:48:49.839 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c'}} 2024-05-01T12:48:49.839 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c'} 2024-05-01T12:48:49.839 INFO:teuthology.task.install:Using flavor: default 2024-05-01T12:48:49.845 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-01T12:48:49.845 INFO:teuthology.task.install:extra packages: [] 2024-05-01T12:48:49.846 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c', 'tag': None, 'wait_for_package': False} 2024-05-01T12:48:49.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=abc5b10088b379036a15386486ef8fee07d8cc3c 2024-05-01T12:48:49.848 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c', 'tag': None, 'wait_for_package': False} 2024-05-01T12:48:49.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=abc5b10088b379036a15386486ef8fee07d8cc3c 2024-05-01T12:48:49.988 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/ 2024-05-01T12:48:49.988 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3122.gabc5b100 2024-05-01T12:48:50.071 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/ 2024-05-01T12:48:50.071 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3122.gabc5b100 2024-05-01T12:48:50.103 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-01T12:48:50.103 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:48:50.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-01T12:48:50.129 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-01T12:48:50.129 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/abc5b10088b379036a15386486ef8fee07d8cc3c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-01T12:48:50.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-01T12:48:50.247 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-alimasa-bn-test/abc5b10088b379036a15386486ef8fee07d8cc3c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-01T12:48:50.247 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:48:50.247 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-01T12:48:50.269 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-01T12:48:50.274 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-01T12:48:50.274 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/abc5b10088b379036a15386486ef8fee07d8cc3c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-01T12:48:50.334 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-05-01T12:48:50.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-05-01T12:48:50.340 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-01T12:48:50.415 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-01T12:48:50.479 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-05-01T12:48:50.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-05-01T12:48:50.571 INFO:teuthology.orchestra.run.smithi050.stdout:50 files removed 2024-05-01T12:48:50.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-05-01T12:48:50.716 INFO:teuthology.orchestra.run.smithi063.stdout:50 files removed 2024-05-01T12:48:50.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-05-01T12:48:51.289 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 200 kB/s | 84 kB 00:00 2024-05-01T12:48:51.367 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 239 kB/s | 84 kB 00:00 2024-05-01T12:48:51.793 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-01T12:48:51.915 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 20 kB/s | 12 kB 00:00 2024-05-01T12:48:52.210 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-05-01T12:48:52.307 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-05-01T12:48:53.874 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-05-01T12:48:54.160 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-05-01T12:48:56.086 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-01T12:48:57.783 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 19 MB 00:02 2024-05-01T12:49:00.104 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.2 MB 00:00 2024-05-01T12:49:01.889 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-05-01T12:49:01.901 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-01T12:49:03.360 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-05-01T12:49:05.006 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-05-01T12:49:06.887 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-05-01T12:49:10.248 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-01T12:49:12.095 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-05-01T12:49:12.477 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:49:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:12.480 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:49:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 11 M 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.4 M 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.1 M 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 5.4 M 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 21 M 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 25 k 2024-05-01T12:49:12.481 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 746 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 500 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 5.4 M 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-01T12:49:12.482 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 45 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 135 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 162 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 320 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 299 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 99 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:12.483 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:Install 30 Packages 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 60 M 2024-05-01T12:49:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:49:13.874 INFO:teuthology.orchestra.run.smithi063.stdout:(1/32): ceph-base-19.0.0-3122.gabc5b100.el9.x86 5.6 MB/s | 5.4 MB 00:00 2024-05-01T12:49:13.974 INFO:teuthology.orchestra.run.smithi063.stdout:(2/32): ceph-selinux-19.0.0-3122.gabc5b100.el9. 251 kB/s | 25 kB 00:00 2024-05-01T12:49:14.332 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 11 M 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.4 M 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.1 M 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 5.4 M 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 21 M 2024-05-01T12:49:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 25 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 746 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 500 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 5.4 M 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-01T12:49:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 45 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 135 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 162 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 320 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 299 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 99 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:49:14.337 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:Install 30 Packages 2024-05-01T12:49:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-05-01T12:49:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 60 M 2024-05-01T12:49:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:49:14.416 INFO:teuthology.orchestra.run.smithi063.stdout:(3/32): ceph-radosgw-19.0.0-3122.gabc5b100.el9. 7.1 MB/s | 11 MB 00:01 2024-05-01T12:49:14.517 INFO:teuthology.orchestra.run.smithi063.stdout:(4/32): libradosstriper1-19.0.0-3122.gabc5b100. 4.9 MB/s | 500 kB 00:00 2024-05-01T12:49:14.616 INFO:teuthology.orchestra.run.smithi063.stdout:(5/32): libcephfs2-19.0.0-3122.gabc5b100.el9.x8 1.1 MB/s | 746 kB 00:00 2024-05-01T12:49:14.709 INFO:teuthology.orchestra.run.smithi063.stdout:(6/32): python3-ceph-argparse-19.0.0-3122.gabc5 492 kB/s | 45 kB 00:00 2024-05-01T12:49:14.800 INFO:teuthology.orchestra.run.smithi063.stdout:(7/32): python3-ceph-common-19.0.0-3122.gabc5b1 1.4 MB/s | 135 kB 00:00 2024-05-01T12:49:15.067 INFO:teuthology.orchestra.run.smithi063.stdout:(8/32): ceph-common-19.0.0-3122.gabc5b100.el9.x 9.8 MB/s | 21 MB 00:02 2024-05-01T12:49:15.092 INFO:teuthology.orchestra.run.smithi063.stdout:(9/32): python3-cephfs-19.0.0-3122.gabc5b100.el 556 kB/s | 162 kB 00:00 2024-05-01T12:49:15.176 INFO:teuthology.orchestra.run.smithi063.stdout:(10/32): librgw2-19.0.0-3122.gabc5b100.el9.x86_ 8.1 MB/s | 5.4 MB 00:00 2024-05-01T12:49:15.201 INFO:teuthology.orchestra.run.smithi063.stdout:(11/32): python3-rbd-19.0.0-3122.gabc5b100.el9. 2.7 MB/s | 299 kB 00:00 2024-05-01T12:49:15.260 INFO:teuthology.orchestra.run.smithi063.stdout:(12/32): python3-rgw-19.0.0-3122.gabc5b100.el9. 1.2 MB/s | 99 kB 00:00 2024-05-01T12:49:15.419 INFO:teuthology.orchestra.run.smithi063.stdout:(13/32): python3-rados-19.0.0-3122.gabc5b100.el 913 kB/s | 320 kB 00:00 2024-05-01T12:49:15.952 INFO:teuthology.orchestra.run.smithi063.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 142 kB/s | 107 kB 00:00 2024-05-01T12:49:15.963 INFO:teuthology.orchestra.run.smithi050.stdout:(1/32): ceph-radosgw-19.0.0-3122.gabc5b100.el9. 7.7 MB/s | 11 MB 00:01 2024-05-01T12:49:16.030 INFO:teuthology.orchestra.run.smithi050.stdout:(2/32): ceph-selinux-19.0.0-3122.gabc5b100.el9. 380 kB/s | 25 kB 00:00 2024-05-01T12:49:16.061 INFO:teuthology.orchestra.run.smithi063.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 200 kB/s | 160 kB 00:00 2024-05-01T12:49:16.078 INFO:teuthology.orchestra.run.smithi063.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 69 kB/s | 45 kB 00:00 2024-05-01T12:49:16.130 INFO:teuthology.orchestra.run.smithi050.stdout:(3/32): libcephfs2-19.0.0-3122.gabc5b100.el9.x8 7.3 MB/s | 746 kB 00:00 2024-05-01T12:49:16.239 INFO:teuthology.orchestra.run.smithi050.stdout:(4/32): libradosstriper1-19.0.0-3122.gabc5b100. 4.5 MB/s | 500 kB 00:00 2024-05-01T12:49:16.458 INFO:teuthology.orchestra.run.smithi050.stdout:(5/32): ceph-common-19.0.0-3122.gabc5b100.el9.x 11 MB/s | 21 MB 00:01 2024-05-01T12:49:16.523 INFO:teuthology.orchestra.run.smithi050.stdout:(6/32): python3-ceph-argparse-19.0.0-3122.gabc5 681 kB/s | 45 kB 00:00 2024-05-01T12:49:16.615 INFO:teuthology.orchestra.run.smithi050.stdout:(7/32): librgw2-19.0.0-3122.gabc5b100.el9.x86_6 14 MB/s | 5.4 MB 00:00 2024-05-01T12:49:16.640 INFO:teuthology.orchestra.run.smithi050.stdout:(8/32): python3-ceph-common-19.0.0-3122.gabc5b1 1.1 MB/s | 135 kB 00:00 2024-05-01T12:49:16.720 INFO:teuthology.orchestra.run.smithi063.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 66 kB/s | 42 kB 00:00 2024-05-01T12:49:16.732 INFO:teuthology.orchestra.run.smithi050.stdout:(9/32): ceph-base-19.0.0-3122.gabc5b100.el9.x86 2.5 MB/s | 5.4 MB 00:02 2024-05-01T12:49:16.757 INFO:teuthology.orchestra.run.smithi050.stdout:(10/32): python3-cephfs-19.0.0-3122.gabc5b100.e 1.1 MB/s | 162 kB 00:00 2024-05-01T12:49:16.782 INFO:teuthology.orchestra.run.smithi050.stdout:(11/32): python3-rados-19.0.0-3122.gabc5b100.el 2.2 MB/s | 320 kB 00:00 2024-05-01T12:49:16.807 INFO:teuthology.orchestra.run.smithi050.stdout:(12/32): python3-rgw-19.0.0-3122.gabc5b100.el9. 1.9 MB/s | 99 kB 00:00 2024-05-01T12:49:16.833 INFO:teuthology.orchestra.run.smithi050.stdout:(13/32): python3-rbd-19.0.0-3122.gabc5b100.el9. 2.9 MB/s | 299 kB 00:00 2024-05-01T12:49:16.895 INFO:teuthology.orchestra.run.smithi063.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 703 kB/s | 662 kB 00:00 2024-05-01T12:49:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 341 kB/s | 292 kB 00:00 2024-05-01T12:49:16.942 INFO:teuthology.orchestra.run.smithi050.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-05-01T12:49:16.967 INFO:teuthology.orchestra.run.smithi050.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-05-01T12:49:16.992 INFO:teuthology.orchestra.run.smithi050.stdout:(16/32): boost-program-options-1.75.0-8.el9.x86 510 kB/s | 107 kB 00:00 2024-05-01T12:49:17.059 INFO:teuthology.orchestra.run.smithi050.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-05-01T12:49:17.084 INFO:teuthology.orchestra.run.smithi050.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-05-01T12:49:17.110 INFO:teuthology.orchestra.run.smithi050.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-05-01T12:49:17.135 INFO:teuthology.orchestra.run.smithi050.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-05-01T12:49:17.160 INFO:teuthology.orchestra.run.smithi050.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-05-01T12:49:17.185 INFO:teuthology.orchestra.run.smithi050.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-05-01T12:49:17.244 INFO:teuthology.orchestra.run.smithi050.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 448 kB/s | 26 kB 00:00 2024-05-01T12:49:17.295 INFO:teuthology.orchestra.run.smithi050.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-05-01T12:49:17.320 INFO:teuthology.orchestra.run.smithi050.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-05-01T12:49:17.345 INFO:teuthology.orchestra.run.smithi050.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-01T12:49:17.429 INFO:teuthology.orchestra.run.smithi050.stdout:(27/32): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-01T12:49:17.454 INFO:teuthology.orchestra.run.smithi050.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-05-01T12:49:17.488 INFO:teuthology.orchestra.run.smithi050.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-05-01T12:49:17.529 INFO:teuthology.orchestra.run.smithi063.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 57 kB/s | 36 kB 00:00 2024-05-01T12:49:17.538 INFO:teuthology.orchestra.run.smithi050.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-05-01T12:49:17.596 INFO:teuthology.orchestra.run.smithi063.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 70 kB/s | 47 kB 00:00 2024-05-01T12:49:17.621 INFO:teuthology.orchestra.run.smithi063.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 245 kB/s | 221 kB 00:00 2024-05-01T12:49:17.896 INFO:teuthology.orchestra.run.smithi063.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-05-01T12:49:17.988 INFO:teuthology.orchestra.run.smithi063.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 673 kB/s | 309 kB 00:00 2024-05-01T12:49:18.013 INFO:teuthology.orchestra.run.smithi063.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-05-01T12:49:18.139 INFO:teuthology.orchestra.run.smithi063.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 448 kB/s | 67 kB 00:00 2024-05-01T12:49:18.171 INFO:teuthology.orchestra.run.smithi050.stdout:(31/32): librbd1-19.0.0-3122.gabc5b100.el9.x86_ 4.6 MB/s | 3.1 MB 00:00 2024-05-01T12:49:18.272 INFO:teuthology.orchestra.run.smithi063.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.2 MB/s | 839 kB 00:00 2024-05-01T12:49:18.314 INFO:teuthology.orchestra.run.smithi050.stdout:(32/32): librados2-19.0.0-3122.gabc5b100.el9.x8 4.0 MB/s | 3.4 MB 00:00 2024-05-01T12:49:18.314 INFO:teuthology.orchestra.run.smithi063.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-05-01T12:49:18.316 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:49:18.316 INFO:teuthology.orchestra.run.smithi050.stdout:Total 15 MB/s | 60 MB 00:03 2024-05-01T12:49:18.381 INFO:teuthology.orchestra.run.smithi063.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-05-01T12:49:18.422 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:49:18.464 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:49:18.464 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:49:18.865 INFO:teuthology.orchestra.run.smithi063.stdout:(30/32): librbd1-19.0.0-3122.gabc5b100.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-05-01T12:49:18.957 INFO:teuthology.orchestra.run.smithi063.stdout:(31/32): librados2-19.0.0-3122.gabc5b100.el9.x8 5.3 MB/s | 3.4 MB 00:00 2024-05-01T12:49:18.958 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:49:18.958 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:49:19.265 INFO:teuthology.orchestra.run.smithi063.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-05-01T12:49:19.268 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:49:19.268 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.9 MB/s | 60 MB 00:06 2024-05-01T12:49:19.374 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:49:19.416 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:49:19.416 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:49:19.910 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:49:19.910 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:49:20.497 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:49:20.582 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-01T12:49:20.610 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-01T12:49:20.636 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-01T12:49:20.659 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-01T12:49:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-01T12:49:20.878 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-01T12:49:20.941 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-01T12:49:21.351 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-01T12:49:21.404 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:21.405 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:49:21.483 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-01T12:49:21.489 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:21.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-01T12:49:21.545 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-01T12:49:21.568 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-01T12:49:21.581 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 8/34 2024-05-01T12:49:21.591 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-01T12:49:21.605 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 9/34 2024-05-01T12:49:21.652 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 9/34 2024-05-01T12:49:21.681 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-01T12:49:21.716 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9. 11/34 2024-05-01T12:49:21.769 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-01T12:49:21.786 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 12/34 2024-05-01T12:49:21.808 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 13/34 2024-05-01T12:49:21.824 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-01T12:49:21.873 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 13/34 2024-05-01T12:49:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-01T12:49:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-01T12:49:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-01T12:49:22.090 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-01T12:49:22.123 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-01T12:49:22.152 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-01T12:49:22.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-01T12:49:22.184 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-01T12:49:22.272 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:22.365 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:22.454 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 8/34 2024-05-01T12:49:22.490 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 9/34 2024-05-01T12:49:22.514 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-01T12:49:22.553 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 9/34 2024-05-01T12:49:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 22/34 2024-05-01T12:49:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-01T12:49:22.625 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9. 11/34 2024-05-01T12:49:22.647 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 22/34 2024-05-01T12:49:22.677 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 23/34 2024-05-01T12:49:22.695 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 12/34 2024-05-01T12:49:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-01T12:49:22.717 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 13/34 2024-05-01T12:49:22.781 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 13/34 2024-05-01T12:49:22.815 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-01T12:49:22.927 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-01T12:49:22.958 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-01T12:49:22.998 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-01T12:49:23.031 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-01T12:49:23.060 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-01T12:49:23.093 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-01T12:49:23.219 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-01T12:49:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 26/34 2024-05-01T12:49:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 26/34 2024-05-01T12:49:23.385 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 27/34 2024-05-01T12:49:23.419 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x8 28/34 2024-05-01T12:49:23.422 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-01T12:49:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 22/34 2024-05-01T12:49:23.564 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 22/34 2024-05-01T12:49:23.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 23/34 2024-05-01T12:49:23.630 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-01T12:49:24.136 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-01T12:49:24.166 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 26/34 2024-05-01T12:49:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 26/34 2024-05-01T12:49:24.319 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 27/34 2024-05-01T12:49:24.352 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x8 28/34 2024-05-01T12:49:25.797 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:26.405 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:26.445 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 30/34 2024-05-01T12:49:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 30/34 2024-05-01T12:49:26.520 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-01T12:49:26.520 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-01T12:49:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:26.546 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:27.272 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 29/34 2024-05-01T12:49:27.311 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 30/34 2024-05-01T12:49:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 30/34 2024-05-01T12:49:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-01T12:49:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-01T12:49:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-05-01T12:49:36.567 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-05-01T12:49:36.568 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-05-01T12:49:36.568 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-05-01T12:49:36.568 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-05-01T12:49:37.418 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-05-01T12:49:37.419 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-05-01T12:49:37.419 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-05-01T12:49:37.419 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 32/34 2024-05-01T12:49:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 32/34 2024-05-01T12:49:37.592 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:49:37.592 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-01T12:49:37.593 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-01T12:49:37.593 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-01T12:49:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-01T12:49:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-01T12:49:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:38.421 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 32/34 2024-05-01T12:49:38.446 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 32/34 2024-05-01T12:49:38.446 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:49:38.446 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-01T12:49:38.447 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-01T12:49:38.447 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-01T12:49:38.447 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:38.449 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-01T12:49:38.531 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-01T12:49:38.532 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 1/34 2024-05-01T12:49:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 3/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 4/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 5/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 6/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9. 8/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x8 9/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 10/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 11/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 12/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 13/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-01T12:49:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-01T12:49:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-01T12:49:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 33/34 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-01T12:49:39.589 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.589 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-01T12:49:39.589 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-01T12:49:39.589 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:39.590 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:49:39.659 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:39.659 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 1/34 2024-05-01T12:49:39.659 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 3/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 4/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 5/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_6 6/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 7/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9. 8/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x8 9/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 10/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 11/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 12/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 13/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-01T12:49:39.660 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-01T12:49:39.661 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-01T12:49:39.662 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-01T12:49:39.662 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-01T12:49:39.662 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 31/34 2024-05-01T12:49:39.662 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-01T12:49:39.662 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 33/34 2024-05-01T12:49:39.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-05-01T12:49:40.401 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:28 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-01T12:49:40.465 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:49:40.526 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:49:40.526 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:40.526 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 49 M 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 163 k 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-05-01T12:49:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:40.528 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 50 M 2024-05-01T12:49:40.528 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 211 M 2024-05-01T12:49:40.528 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:49:40.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-05-01T12:49:40.970 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): libcephsqlite-19.0.0-3122.gabc5b100.el9. 431 kB/s | 163 kB 00:00 2024-05-01T12:49:41.320 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:31 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:49:41.444 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 49 M 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:49:41.445 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 163 k 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 50 M 2024-05-01T12:49:41.447 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 211 M 2024-05-01T12:49:41.447 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:49:41.466 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 348 kB/s | 305 kB 00:00 2024-05-01T12:49:41.681 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 90 kB/s | 64 kB 00:00 2024-05-01T12:49:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-05-01T12:49:41.955 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 951 kB/s | 64 kB 00:00 2024-05-01T12:49:42.367 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): libcephsqlite-19.0.0-3122.gabc5b100.el9. 214 kB/s | 163 kB 00:00 2024-05-01T12:49:43.122 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): ceph-test-19.0.0-3122.gabc5b100.el9.x86_ 19 MB/s | 49 MB 00:02 2024-05-01T12:49:43.125 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:49:43.125 INFO:teuthology.orchestra.run.smithi050.stdout:Total 19 MB/s | 50 MB 00:02 2024-05-01T12:49:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:49:43.176 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:49:43.176 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:49:43.567 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:49:43.567 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:49:44.116 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:49:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-01T12:49:44.247 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-01T12:49:44.275 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 3/4 2024-05-01T12:49:47.173 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): ceph-test-19.0.0-3122.gabc5b100.el9.x86_ 8.8 MB/s | 49 MB 00:05 2024-05-01T12:49:47.175 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:49:47.175 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.7 MB/s | 50 MB 00:05 2024-05-01T12:49:47.203 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:49:47.225 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:49:47.226 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:49:47.616 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:49:47.616 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:49:48.184 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:49:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-01T12:49:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-01T12:49:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 3/4 2024-05-01T12:49:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 3/4 2024-05-01T12:49:49.225 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 4/4 2024-05-01T12:49:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 4/4 2024-05-01T12:49:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 1/4 2024-05-01T12:49:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 2/4 2024-05-01T12:49:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-01T12:49:51.890 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-01T12:49:51.890 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:51.890 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:49:52.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-05-01T12:49:52.726 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:40 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:49:52.857 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:49:52.859 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:49:52.859 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 6.6 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 2.4 M 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 1.5 M 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 249 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 4.7 M 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 17 M 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-01T12:49:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-01T12:49:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:49:52.862 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:49:52.863 INFO:teuthology.orchestra.run.smithi050.stdout:Install 36 Packages 2024-05-01T12:49:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:49:52.863 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 30 M 2024-05-01T12:49:52.863 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 107 M 2024-05-01T12:49:52.863 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:49:53.200 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 3/4 2024-05-01T12:49:53.274 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 4/4 2024-05-01T12:49:53.675 INFO:teuthology.orchestra.run.smithi050.stdout:(1/36): ceph-19.0.0-3122.gabc5b100.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-05-01T12:49:53.733 INFO:teuthology.orchestra.run.smithi050.stdout:(2/36): ceph-mds-19.0.0-3122.gabc5b100.el9.x86_ 6.0 MB/s | 2.4 MB 00:00 2024-05-01T12:49:53.775 INFO:teuthology.orchestra.run.smithi050.stdout:(3/36): ceph-mgr-19.0.0-3122.gabc5b100.el9.x86_ 3.4 MB/s | 1.5 MB 00:00 2024-05-01T12:49:53.884 INFO:teuthology.orchestra.run.smithi050.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3122.gabc5 2.2 MB/s | 249 kB 00:00 2024-05-01T12:49:54.518 INFO:teuthology.orchestra.run.smithi050.stdout:(5/36): ceph-osd-19.0.0-3122.gabc5b100.el9.x86_ 21 MB/s | 17 MB 00:00 2024-05-01T12:49:54.601 INFO:teuthology.orchestra.run.smithi050.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 353 kB/s | 253 kB 00:00 2024-05-01T12:49:54.718 INFO:teuthology.orchestra.run.smithi050.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 914 kB/s | 106 kB 00:00 2024-05-01T12:49:54.835 INFO:teuthology.orchestra.run.smithi050.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-05-01T12:49:54.952 INFO:teuthology.orchestra.run.smithi050.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.0 MB/s | 125 kB 00:00 2024-05-01T12:49:55.161 INFO:teuthology.orchestra.run.smithi050.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-05-01T12:49:55.486 INFO:teuthology.orchestra.run.smithi050.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-05-01T12:49:55.622 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 4/4 2024-05-01T12:49:55.622 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 1/4 2024-05-01T12:49:55.622 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 2/4 2024-05-01T12:49:55.622 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-01T12:49:55.694 INFO:teuthology.orchestra.run.smithi050.stdout:(12/36): ceph-mon-19.0.0-3122.gabc5b100.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-01T12:49:55.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:55.907 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:49:56.078 INFO:teuthology.orchestra.run.smithi050.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 205 kB/s | 188 kB 00:00 2024-05-01T12:49:56.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-05-01T12:49:56.428 INFO:teuthology.orchestra.run.smithi050.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 183 kB/s | 172 kB 00:00 2024-05-01T12:49:56.453 INFO:teuthology.orchestra.run.smithi050.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 46 kB/s | 35 kB 00:00 2024-05-01T12:49:56.579 INFO:teuthology.orchestra.run.smithi050.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-05-01T12:49:56.694 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:46 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:49:56.820 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:49:56.822 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:56.822 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:49:56.822 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 6.6 k 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 2.4 M 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 1.5 M 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 249 k 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 4.7 M 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 17 M 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-01T12:49:56.823 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-01T12:49:56.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-01T12:49:56.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout:Install 36 Packages 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:49:56.826 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 30 M 2024-05-01T12:49:56.827 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 107 M 2024-05-01T12:49:56.827 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:49:56.837 INFO:teuthology.orchestra.run.smithi050.stdout:(17/36): python3-pytz-2021.1-5.el9.noarch.rpm 67 kB/s | 51 kB 00:00 2024-05-01T12:49:57.204 INFO:teuthology.orchestra.run.smithi050.stdout:(18/36): python3-toml-0.10.2-6.el9.noarch.rpm 54 kB/s | 42 kB 00:00 2024-05-01T12:49:57.382 INFO:teuthology.orchestra.run.smithi063.stdout:(1/36): ceph-19.0.0-3122.gabc5b100.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-05-01T12:49:57.424 INFO:teuthology.orchestra.run.smithi063.stdout:(2/36): ceph-mgr-19.0.0-3122.gabc5b100.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-05-01T12:49:57.507 INFO:teuthology.orchestra.run.smithi063.stdout:(3/36): ceph-mds-19.0.0-3122.gabc5b100.el9.x86_ 5.4 MB/s | 2.4 MB 00:00 2024-05-01T12:49:57.591 INFO:teuthology.orchestra.run.smithi063.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3122.gabc5 2.9 MB/s | 249 kB 00:00 2024-05-01T12:49:57.741 INFO:teuthology.orchestra.run.smithi063.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-05-01T12:49:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 7.0 MB/s | 1.2 MB 00:00 2024-05-01T12:49:57.959 INFO:teuthology.orchestra.run.smithi063.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-01T12:49:58.009 INFO:teuthology.orchestra.run.smithi063.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-05-01T12:49:58.059 INFO:teuthology.orchestra.run.smithi063.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-05-01T12:49:58.109 INFO:teuthology.orchestra.run.smithi063.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-05-01T12:49:58.326 INFO:teuthology.orchestra.run.smithi063.stdout:(11/36): ceph-osd-19.0.0-3122.gabc5b100.el9.x86 19 MB/s | 17 MB 00:00 2024-05-01T12:49:58.821 INFO:teuthology.orchestra.run.smithi063.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 266 kB/s | 188 kB 00:00 2024-05-01T12:49:58.902 INFO:teuthology.orchestra.run.smithi063.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 299 kB/s | 172 kB 00:00 2024-05-01T12:49:58.927 INFO:teuthology.orchestra.run.smithi063.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-05-01T12:49:59.019 INFO:teuthology.orchestra.run.smithi063.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-05-01T12:49:59.044 INFO:teuthology.orchestra.run.smithi063.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 356 kB/s | 42 kB 00:00 2024-05-01T12:49:59.094 INFO:teuthology.orchestra.run.smithi063.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-05-01T12:49:59.486 INFO:teuthology.orchestra.run.smithi063.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-05-01T12:49:59.587 INFO:teuthology.orchestra.run.smithi063.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-05-01T12:49:59.663 INFO:teuthology.orchestra.run.smithi063.stdout:(20/36): ceph-mon-19.0.0-3122.gabc5b100.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-05-01T12:49:59.754 INFO:teuthology.orchestra.run.smithi063.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-05-01T12:49:59.821 INFO:teuthology.orchestra.run.smithi063.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-05-01T12:49:59.888 INFO:teuthology.orchestra.run.smithi063.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 170 kB/s | 18 kB 00:00 2024-05-01T12:49:59.921 INFO:teuthology.orchestra.run.smithi063.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-05-01T12:49:59.929 INFO:teuthology.orchestra.run.smithi050.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 14 kB/s | 43 kB 00:03 2024-05-01T12:49:59.980 INFO:teuthology.orchestra.run.smithi063.stdout:(25/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 34 kB/s | 11 kB 00:00 2024-05-01T12:50:00.005 INFO:teuthology.orchestra.run.smithi063.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-05-01T12:50:00.031 INFO:teuthology.orchestra.run.smithi063.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-05-01T12:50:00.088 INFO:teuthology.orchestra.run.smithi050.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 60 kB/s | 172 kB 00:02 2024-05-01T12:50:00.131 INFO:teuthology.orchestra.run.smithi063.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 630 kB/s | 79 kB 00:00 2024-05-01T12:50:00.164 INFO:teuthology.orchestra.run.smithi063.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-05-01T12:50:00.171 INFO:teuthology.orchestra.run.smithi050.stdout:(21/36): luarocks-3.9.2-1.el9.noarch.rpm 42 kB/s | 151 kB 00:03 2024-05-01T12:50:00.190 INFO:teuthology.orchestra.run.smithi063.stdout:(30/36): python3-logutils-0.3.5-21.el9.noarch.r 221 kB/s | 46 kB 00:00 2024-05-01T12:50:00.197 INFO:teuthology.orchestra.run.smithi050.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-05-01T12:50:00.231 INFO:teuthology.orchestra.run.smithi063.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2024-05-01T12:50:00.272 INFO:teuthology.orchestra.run.smithi050.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-05-01T12:50:00.273 INFO:teuthology.orchestra.run.smithi063.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 826 kB/s | 90 kB 00:00 2024-05-01T12:50:00.297 INFO:teuthology.orchestra.run.smithi050.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-05-01T12:50:00.315 INFO:teuthology.orchestra.run.smithi063.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-05-01T12:50:00.372 INFO:teuthology.orchestra.run.smithi050.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-05-01T12:50:00.398 INFO:teuthology.orchestra.run.smithi050.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-05-01T12:50:00.399 INFO:teuthology.orchestra.run.smithi063.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-05-01T12:50:00.431 INFO:teuthology.orchestra.run.smithi050.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 714 kB/s | 358 kB 00:00 2024-05-01T12:50:00.432 INFO:teuthology.orchestra.run.smithi063.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-05-01T12:50:00.466 INFO:teuthology.orchestra.run.smithi063.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-05-01T12:50:00.472 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:00.472 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.3 MB/s | 30 MB 00:03 2024-05-01T12:50:00.616 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:00.660 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:00.660 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:00.975 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:00.976 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:01.398 INFO:teuthology.orchestra.run.smithi050.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 79 kB/s | 79 kB 00:01 2024-05-01T12:50:01.424 INFO:teuthology.orchestra.run.smithi050.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 44 kB/s | 46 kB 00:01 2024-05-01T12:50:01.449 INFO:teuthology.orchestra.run.smithi050.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 268 kB/s | 272 kB 00:01 2024-05-01T12:50:01.657 INFO:teuthology.orchestra.run.smithi050.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 172 kB/s | 36 kB 00:00 2024-05-01T12:50:01.683 INFO:teuthology.orchestra.run.smithi050.stdout:(32/36): python3-portend-3.1.0-2.el9.noarch.rpm 58 kB/s | 16 kB 00:00 2024-05-01T12:50:01.708 INFO:teuthology.orchestra.run.smithi050.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 316 kB/s | 90 kB 00:00 2024-05-01T12:50:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:02.302 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-01T12:50:02.335 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-01T12:50:02.363 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-01T12:50:02.439 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-01T12:50:02.488 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-01T12:50:02.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-01T12:50:02.618 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-01T12:50:02.693 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-01T12:50:02.749 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-01T12:50:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-01T12:50:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-01T12:50:02.847 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-01T12:50:02.880 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-01T12:50:02.921 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-01T12:50:02.965 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-01T12:50:02.967 INFO:teuthology.orchestra.run.smithi050.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 16 kB/s | 20 kB 00:01 2024-05-01T12:50:02.998 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-01T12:50:03.033 INFO:teuthology.orchestra.run.smithi050.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 316 kB/s | 427 kB 00:01 2024-05-01T12:50:03.037 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-01T12:50:03.092 INFO:teuthology.orchestra.run.smithi050.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 160 kB/s | 230 kB 00:01 2024-05-01T12:50:03.095 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:03.095 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.9 MB/s | 30 MB 00:10 2024-05-01T12:50:03.114 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-01T12:50:03.245 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:03.265 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-01T12:50:03.289 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:03.289 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:03.356 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-01T12:50:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-01T12:50:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-01T12:50:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-01T12:50:03.577 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-01T12:50:03.602 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:03.603 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:03.775 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-01T12:50:03.824 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-01T12:50:03.907 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-01T12:50:04.004 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-01T12:50:04.048 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-01T12:50:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-01T12:50:04.345 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9. 31/36 2024-05-01T12:50:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 32/36 2024-05-01T12:50:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 32/36 2024-05-01T12:50:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-01T12:50:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-01T12:50:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-01T12:50:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:04.937 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-01T12:50:04.970 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-01T12:50:04.998 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-01T12:50:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-01T12:50:05.122 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-01T12:50:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-01T12:50:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-01T12:50:05.338 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-01T12:50:05.384 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-01T12:50:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-01T12:50:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-01T12:50:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-01T12:50:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-01T12:50:05.548 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-01T12:50:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-01T12:50:05.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-01T12:50:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-01T12:50:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-01T12:50:05.895 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-01T12:50:05.941 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 33/36 2024-05-01T12:50:05.970 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 33/36 2024-05-01T12:50:05.970 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:05.970 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-01T12:50:05.970 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-01T12:50:05.971 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-01T12:50:05.971 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-01T12:50:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-01T12:50:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-01T12:50:06.282 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-01T12:50:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-01T12:50:06.434 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 34/36 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 34/36 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-01T12:50:06.459 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:06.545 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-01T12:50:06.594 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-01T12:50:06.675 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-01T12:50:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 35/36 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 35/36 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-01T12:50:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:06.774 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 36/36 2024-05-01T12:50:06.774 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-01T12:50:06.818 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-01T12:50:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-01T12:50:07.107 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9. 31/36 2024-05-01T12:50:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 32/36 2024-05-01T12:50:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 32/36 2024-05-01T12:50:07.163 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:07.164 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-01T12:50:07.164 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-01T12:50:07.164 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-01T12:50:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:08.409 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 36/36 2024-05-01T12:50:08.409 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 1/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 2/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 3/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 4/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 5/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9. 6/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-01T12:50:08.410 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-01T12:50:08.411 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-01T12:50:08.412 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-01T12:50:08.412 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-01T12:50:08.412 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-01T12:50:08.412 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-01T12:50:08.413 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-01T12:50:08.413 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-01T12:50:08.695 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 33/36 2024-05-01T12:50:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 33/36 2024-05-01T12:50:08.723 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:08.723 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-01T12:50:08.724 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-01T12:50:08.724 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-01T12:50:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:09.196 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 34/36 2024-05-01T12:50:09.220 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 34/36 2024-05-01T12:50:09.220 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:09.220 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-01T12:50:09.220 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-01T12:50:09.221 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-01T12:50:09.221 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-01T12:50:09.321 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-01T12:50:09.322 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-01T12:50:09.323 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-01T12:50:09.324 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-01T12:50:09.324 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:09.324 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:09.455 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 35/36 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 35/36 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-01T12:50:09.479 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:09.553 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 36/36 2024-05-01T12:50:09.653 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-05-01T12:50:10.192 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:00 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:10.274 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:10.320 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:10.321 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:10.321 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:10.393 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-05-01T12:50:10.920 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:00 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:11.045 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 762 k 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:11.046 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 762 k 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 769 k 2024-05-01T12:50:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:11.158 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 36/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 1/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 2/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 3/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 4/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 5/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9. 6/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-01T12:50:11.159 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-01T12:50:11.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-01T12:50:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-01T12:50:11.633 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-19.0.0-3122.gabc5b100.el9.noarch.rpm 1.3 MB/s | 762 kB 00:00 2024-05-01T12:50:11.634 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:11.634 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.3 MB/s | 762 kB 00:00 2024-05-01T12:50:11.634 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:11.640 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:11.640 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:11.646 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:11.646 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:11.728 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:12.193 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-01T12:50:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-01T12:50:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-01T12:50:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-01T12:50:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-01T12:50:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-01T12:50:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:12.228 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:12.287 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:12.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-05-01T12:50:12.615 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:12.839 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:13.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-05-01T12:50:13.068 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:01 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:13.150 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:13.194 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:13.195 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:50:13.195 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:13.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-05-01T12:50:13.581 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:03 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:13.708 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 145 k 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:13.709 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 145 k 2024-05-01T12:50:13.710 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 435 k 2024-05-01T12:50:13.710 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:13.798 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:01 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:13.923 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 762 k 2024-05-01T12:50:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 762 k 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 769 k 2024-05-01T12:50:13.925 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:50:14.149 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-19.0.0-3122.gabc5b1 330 kB/s | 145 kB 00:00 2024-05-01T12:50:14.149 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:14.150 INFO:teuthology.orchestra.run.smithi063.stdout:Total 329 kB/s | 145 kB 00:00 2024-05-01T12:50:14.150 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:14.158 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:14.158 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:14.219 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:14.220 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:14.383 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:14.467 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:14.493 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:14.493 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:14.493 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-01T12:50:14.493 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:14.771 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-19.0.0-3122.gabc5b100.el9.noarch.rpm 902 kB/s | 762 kB 00:00 2024-05-01T12:50:14.771 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:14.772 INFO:teuthology.orchestra.run.smithi050.stdout:Total 900 kB/s | 762 kB 00:00 2024-05-01T12:50:14.772 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:14.779 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:14.779 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:14.785 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:14.785 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:14.863 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:15.305 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:15.355 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:15.356 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:15.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-05-01T12:50:15.733 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/1 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:15.959 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:16.133 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:06 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:16.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-05-01T12:50:16.216 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:16.261 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:16.262 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:16.262 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:16.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-05-01T12:50:16.732 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:04 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:16.859 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 145 k 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 145 k 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 435 k 2024-05-01T12:50:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:50:16.864 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:06 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:16.991 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 3.5 M 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 34 k 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 16 k 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-01T12:50:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-05-01T12:50:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:16.994 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 8.6 M 2024-05-01T12:50:16.994 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 104 M 2024-05-01T12:50:16.994 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:17.079 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-19.0.0-3122.gabc5b1 662 kB/s | 145 kB 00:00 2024-05-01T12:50:17.080 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:17.080 INFO:teuthology.orchestra.run.smithi050.stdout:Total 657 kB/s | 145 kB 00:00 2024-05-01T12:50:17.080 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:17.089 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:17.089 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:17.149 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:17.149 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:17.292 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:17.364 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:17.389 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:17.390 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:17.390 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-01T12:50:17.390 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:17.579 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3122.gabc 97 kB/s | 16 kB 00:00 2024-05-01T12:50:17.595 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3122.gab 184 kB/s | 34 kB 00:00 2024-05-01T12:50:17.846 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-05-01T12:50:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100. 1/1 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:18.252 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:18.279 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-05-01T12:50:18.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-05-01T12:50:18.571 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-01T12:50:18.663 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 747 kB/s | 548 kB 00:00 2024-05-01T12:50:18.872 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): ceph-mgr-dashboard-19.0.0-3122.gabc5b10 2.4 MB/s | 3.5 MB 00:01 2024-05-01T12:50:19.030 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:07 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:19.097 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 333 kB/s | 144 kB 00:00 2024-05-01T12:50:19.112 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:19.122 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-05-01T12:50:19.156 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:19.157 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:50:19.157 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:19.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-05-01T12:50:19.506 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 461 kB/s | 188 kB 00:00 2024-05-01T12:50:19.750 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:07 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:19.877 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:19.878 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:19.878 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:50:19.878 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 3.5 M 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 34 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 16 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-01T12:50:19.879 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 8.6 M 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 104 M 2024-05-01T12:50:19.880 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:50:19.931 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.5 MB/s | 2.0 MB 00:01 2024-05-01T12:50:19.936 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:19.936 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.9 MB/s | 8.6 MB 00:02 2024-05-01T12:50:19.988 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:20.008 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:20.008 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:20.174 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:20.174 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:20.492 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3122.gab 168 kB/s | 34 kB 00:00 2024-05-01T12:50:20.517 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3122.gabc 72 kB/s | 16 kB 00:00 2024-05-01T12:50:20.660 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-05-01T12:50:20.710 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-05-01T12:50:20.717 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:20.791 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-01T12:50:20.802 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): ceph-mgr-dashboard-19.0.0-3122.gabc5b10 6.9 MB/s | 3.5 MB 00:00 2024-05-01T12:50:20.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-01T12:50:20.874 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-01T12:50:20.911 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-01T12:50:20.994 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-01T12:50:21.060 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-05-01T12:50:21.119 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-05-01T12:50:21.143 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-01T12:50:21.260 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-01T12:50:21.309 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-01T12:50:21.311 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 912 kB/s | 548 kB 00:00 2024-05-01T12:50:21.336 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-01T12:50:21.369 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9 10/12 2024-05-01T12:50:21.411 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-05-01T12:50:21.495 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 385 kB/s | 144 kB 00:00 2024-05-01T12:50:21.520 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-05-01T12:50:21.637 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-05-01T12:50:21.638 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:21.638 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.9 MB/s | 8.6 MB 00:01 2024-05-01T12:50:21.691 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:21.710 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:21.710 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:21.875 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:21.875 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:22.479 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el 11/12 2024-05-01T12:50:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-01T12:50:22.510 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 12/12 2024-05-01T12:50:22.523 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-01T12:50:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-01T12:50:22.600 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-01T12:50:22.679 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-01T12:50:22.904 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-01T12:50:22.907 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-01T12:50:22.964 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-01T12:50:22.991 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-01T12:50:23.016 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9 10/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 12/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el 1/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 2/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9 3/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-01T12:50:23.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-01T12:50:23.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-01T12:50:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:24.007 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:24.114 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el 11/12 2024-05-01T12:50:24.149 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 12/12 2024-05-01T12:50:24.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-01T12:50:24.762 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:14 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:24.889 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:24.890 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-05-01T12:50:24.890 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:24.890 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-05-01T12:50:24.890 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 7.4 M 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-01T12:50:24.891 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 38 M 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 209 M 2024-05-01T12:50:24.892 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:25.240 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-05-01T12:50:25.299 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2024-05-01T12:50:25.466 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 145 kB/s | 33 kB 00:00 2024-05-01T12:50:25.492 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 12/12 2024-05-01T12:50:25.492 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el 1/12 2024-05-01T12:50:25.492 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noa 2/12 2024-05-01T12:50:25.492 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9 3/12 2024-05-01T12:50:25.492 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-01T12:50:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-01T12:50:25.541 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-05-01T12:50:25.624 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 425 kB/s | 35 kB 00:00 2024-05-01T12:50:25.791 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-01T12:50:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-01T12:50:25.793 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:25.793 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:25.917 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-31 9.1 MB/s | 7.4 MB 00:00 2024-05-01T12:50:26.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-01T12:50:26.134 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-05-01T12:50:26.243 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 751 kB/s | 244 kB 00:00 2024-05-01T12:50:26.410 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 250 kB 00:00 2024-05-01T12:50:26.562 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:14 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:26.685 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.5 MB/s | 4.7 MB 00:01 2024-05-01T12:50:26.688 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 7.4 M 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-01T12:50:26.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:26.691 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-05-01T12:50:26.692 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 209 M 2024-05-01T12:50:26.692 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:50:26.794 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.7 MB/s | 5.1 MB 00:00 2024-05-01T12:50:27.205 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-05-01T12:50:27.239 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-05-01T12:50:27.489 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-05-01T12:50:27.598 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-05-01T12:50:27.652 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-05-01T12:50:27.656 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:27.656 INFO:teuthology.orchestra.run.smithi063.stdout:Total 14 MB/s | 38 MB 00:02 2024-05-01T12:50:27.715 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 302 kB/s | 35 kB 00:00 2024-05-01T12:50:28.015 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:28.032 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-31 7.3 MB/s | 7.4 MB 00:01 2024-05-01T12:50:28.043 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:28.043 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:28.340 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-05-01T12:50:28.477 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:28.478 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:28.565 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 458 kB/s | 244 kB 00:00 2024-05-01T12:50:28.766 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-05-01T12:50:28.891 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.3 MB/s | 5.1 MB 00:00 2024-05-01T12:50:29.313 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:29.366 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-01T12:50:29.459 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-01T12:50:29.563 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-01T12:50:29.601 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-01T12:50:29.786 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-01T12:50:29.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-01T12:50:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-01T12:50:30.325 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 1.8 MB/s | 4.7 MB 00:02 2024-05-01T12:50:30.581 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-01T12:50:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-01T12:50:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-01T12:50:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-01T12:50:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 12/12 2024-05-01T12:50:35.475 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 12/12 2024-05-01T12:50:35.475 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 1/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-01T12:50:35.476 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-01T12:50:35.840 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:35.841 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:36.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-05-01T12:50:36.698 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:26 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:36.830 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 49 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-01T12:50:36.832 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-05-01T12:50:36.833 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:36.834 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.2 M 2024-05-01T12:50:36.834 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-05-01T12:50:36.834 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:37.051 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 258 kB/s | 26 kB 00:00 2024-05-01T12:50:37.076 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 154 kB/s | 19 kB 00:00 2024-05-01T12:50:37.151 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-05-01T12:50:37.176 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-05-01T12:50:37.210 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.7 MB/s | 279 kB 00:00 2024-05-01T12:50:37.235 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-05-01T12:50:37.260 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): ceph-mgr-rook-19.0.0-3122.gabc5b100.el9 159 kB/s | 49 kB 00:00 2024-05-01T12:50:37.360 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-05-01T12:50:37.386 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-05-01T12:50:37.503 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-05-01T12:50:37.528 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 769 kB/s | 206 kB 00:00 2024-05-01T12:50:37.603 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 896 kB/s | 90 kB 00:00 2024-05-01T12:50:37.670 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-05-01T12:50:37.671 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:37.671 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-05-01T12:50:37.736 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:37.846 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:37.847 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:38.362 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:38.534 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-01T12:50:38.590 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-01T12:50:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-01T12:50:38.681 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-01T12:50:38.723 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-01T12:50:38.821 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-01T12:50:38.950 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-01T12:50:39.000 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-01T12:50:39.770 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-01T12:50:39.821 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-01T12:50:39.864 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-01T12:50:39.913 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-01T12:50:39.933 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 13/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 13/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 1/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-01T12:50:40.695 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-01T12:50:40.696 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-01T12:50:40.995 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-01T12:50:40.995 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:40.995 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:40.995 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:40.996 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:41.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-05-01T12:50:41.753 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:31 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:41.878 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 142 k 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:41.879 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 7.0 M 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-05-01T12:50:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:42.347 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-05-01T12:50:42.522 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-05-01T12:50:42.656 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3122.gabc5b100.e 319 kB/s | 142 kB 00:00 2024-05-01T12:50:42.840 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-05-01T12:50:42.865 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-05-01T12:50:42.957 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-05-01T12:50:42.959 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:42.960 INFO:teuthology.orchestra.run.smithi063.stdout:Total 6.5 MB/s | 7.0 MB 00:01 2024-05-01T12:50:43.032 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:43.044 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:43.044 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:43.148 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:43.149 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:43.461 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-01T12:50:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-01T12:50:44.261 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-01T12:50:44.348 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-01T12:50:44.419 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-01T12:50:44.446 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 6/6 2024-05-01T12:50:45.174 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 6/6 2024-05-01T12:50:45.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/6 2024-05-01T12:50:45.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-01T12:50:45.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-01T12:50:45.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-01T12:50:45.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-01T12:50:45.426 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-01T12:50:45.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-01T12:50:45.428 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-01T12:50:45.428 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:45.428 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:45.548 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 992 kB/s | 16 MB 00:16 2024-05-01T12:50:45.550 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:45.550 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.0 MB/s | 38 MB 00:18 2024-05-01T12:50:45.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-05-01T12:50:45.912 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:45.940 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:45.940 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:36 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:46.331 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:46.331 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:46.331 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:46.331 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 890 k 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-05-01T12:50:46.332 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:46.333 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 970 k 2024-05-01T12:50:46.333 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.8 M 2024-05-01T12:50:46.333 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:46.374 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:46.374 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:46.534 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 586 kB/s | 80 kB 00:00 2024-05-01T12:50:47.109 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-19.0.0-3122.gabc5b100.el9.x86_ 1.2 MB/s | 890 kB 00:00 2024-05-01T12:50:47.110 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:47.110 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.2 MB/s | 970 kB 00:00 2024-05-01T12:50:47.123 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:47.147 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:47.147 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:47.231 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:47.231 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:47.404 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-01T12:50:47.419 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-01T12:50:47.552 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-01T12:50:47.557 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-01T12:50:47.598 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-01T12:50:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2/2 2024-05-01T12:50:47.784 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-01T12:50:47.822 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-01T12:50:47.901 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-01T12:50:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2/2 2024-05-01T12:50:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/2 2024-05-01T12:50:48.576 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-01T12:50:48.577 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:48.577 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:48.577 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-01T12:50:48.577 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:48.577 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:48.584 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-01T12:50:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-01T12:50:48.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-05-01T12:50:49.358 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:39 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:49.484 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:49.484 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 273 k 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-05-01T12:50:49.485 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:49.486 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 350 k 2024-05-01T12:50:49.486 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.5 M 2024-05-01T12:50:49.486 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:49.898 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 572 kB/s | 77 kB 00:00 2024-05-01T12:50:50.012 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-01T12:50:50.298 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-volume-19.0.0-3122.gabc5b100.el9.no 510 kB/s | 273 kB 00:00 2024-05-01T12:50:50.299 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:50.299 INFO:teuthology.orchestra.run.smithi063.stdout:Total 430 kB/s | 350 kB 00:00 2024-05-01T12:50:50.312 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:50.323 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:50.323 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:50.352 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:50.353 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:50.514 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:50.706 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-01T12:50:50.743 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2/2 2024-05-01T12:50:50.758 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2/2 2024-05-01T12:50:50.758 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:50:50.759 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-01T12:50:50.759 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-01T12:50:51.231 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 12/12 2024-05-01T12:50:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 1/2 2024-05-01T12:50:51.726 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:51.727 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:51.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-05-01T12:50:52.538 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:42 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:52.663 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 127 k 2024-05-01T12:50:52.664 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 127 k 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 456 k 2024-05-01T12:50:52.665 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:52.878 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-19.0.0-3122.gabc5b100.el9.x86_64 595 kB/s | 127 kB 00:00 2024-05-01T12:50:52.879 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:52.879 INFO:teuthology.orchestra.run.smithi063.stdout:Total 591 kB/s | 127 kB 00:00 2024-05-01T12:50:52.879 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:52.885 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:52.885 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:52.951 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:52.951 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:53.111 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:53.218 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 12/12 2024-05-01T12:50:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b 1/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-01T12:50:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-01T12:50:53.611 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-01T12:50:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:53.825 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:50:53.834 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:53.834 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:53.834 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:53.834 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:53.834 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:53.835 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:54.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-05-01T12:50:54.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-05-01T12:50:54.610 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:44 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:54.692 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:54.728 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:50:54.736 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:54.737 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:54.737 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:54.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-05-01T12:50:54.853 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:50:54.854 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:54.854 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:50:54.854 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 49 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-01T12:50:54.855 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-05-01T12:50:54.856 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:50:55.368 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:45 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:55.383 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 186 kB/s | 19 kB 00:00 2024-05-01T12:50:55.425 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 176 kB/s | 26 kB 00:00 2024-05-01T12:50:55.494 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 32 k 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:55.495 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 32 k 2024-05-01T12:50:55.496 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 142 k 2024-05-01T12:50:55.496 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:50:55.508 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-05-01T12:50:55.534 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-01T12:50:55.576 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-05-01T12:50:55.601 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 279 kB 00:00 2024-05-01T12:50:55.685 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 297 kB/s | 32 kB 00:00 2024-05-01T12:50:55.735 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-05-01T12:50:55.844 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): ceph-mgr-rook-19.0.0-3122.gabc5b100.el9 87 kB/s | 49 kB 00:00 2024-05-01T12:50:55.886 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.0 MB/s | 206 kB 00:00 2024-05-01T12:50:55.936 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-05-01T12:50:55.948 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-19.0.0-3122.gabc5b100.el9.x86_6 70 kB/s | 32 kB 00:00 2024-05-01T12:50:55.949 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:55.950 INFO:teuthology.orchestra.run.smithi063.stdout:Total 70 kB/s | 32 kB 00:00 2024-05-01T12:50:55.950 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:50:55.954 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:50:55.954 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:50:55.957 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:50:55.957 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:50:55.994 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-05-01T12:50:56.046 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:50:56.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:56.504 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:50:56.727 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:56.727 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:50:56.727 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:50:56.727 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:50:56.727 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:56.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-05-01T12:50:57.219 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 704 kB/s | 1.0 MB 00:01 2024-05-01T12:50:57.220 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:50:57.221 INFO:teuthology.orchestra.run.smithi050.stdout:Total 953 kB/s | 2.2 MB 00:02 2024-05-01T12:50:57.292 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:50:57.307 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:50:57.307 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:50:57.401 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:50:57.401 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:50:57.484 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:47 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:57.565 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:57.608 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:57.609 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:57.609 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:57.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-05-01T12:50:57.906 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:50:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-01T12:50:58.139 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-01T12:50:58.196 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-01T12:50:58.220 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:48 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-01T12:50:58.273 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-01T12:50:58.302 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:58.346 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:58.347 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:58.347 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:58.371 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-01T12:50:58.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-05-01T12:50:58.483 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-01T12:50:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-01T12:50:58.968 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:48 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:59.050 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:59.093 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:59.094 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:59.094 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:59.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-05-01T12:50:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-01T12:50:59.347 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-01T12:50:59.380 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-01T12:50:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-01T12:50:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 13/13 2024-05-01T12:50:59.699 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:49 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:50:59.780 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:50:59.823 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:50:59.824 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:50:59.824 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:50:59.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-05-01T12:51:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 13/13 2024-05-01T12:51:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 1/13 2024-05-01T12:51:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-01T12:51:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-01T12:51:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-01T12:51:00.182 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-01T12:51:00.436 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:50 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:51:00.487 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-01T12:51:00.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-01T12:51:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-01T12:51:00.489 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:00.489 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:00.517 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:00.561 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:51:00.562 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:51:00.562 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:51:00.641 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-05-01T12:51:00.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-05-01T12:51:01.177 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:51 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:51:01.254 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:49 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:01.257 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:01.301 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:51:01.302 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-05-01T12:51:01.302 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:51:01.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-05-01T12:51:01.379 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:01.379 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 142 k 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-05-01T12:51:01.381 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:01.673 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.2 MB/s | 248 kB 00:00 2024-05-01T12:51:01.840 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-05-01T12:51:01.865 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3122.gabc5b100.e 470 kB/s | 142 kB 00:00 2024-05-01T12:51:01.906 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:51 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:51:02.031 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 85 k 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 85 k 2024-05-01T12:51:02.032 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 231 k 2024-05-01T12:51:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:51:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 306 kB/s | 51 kB 00:00 2024-05-01T12:51:02.057 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-05-01T12:51:02.149 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-05-01T12:51:02.150 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:02.150 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.1 MB/s | 7.0 MB 00:00 2024-05-01T12:51:02.225 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:02.237 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:02.237 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:02.340 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:02.341 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:02.410 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-19.0.0-3122.gabc5b100.el9.x86_64.rpm 226 kB/s | 85 kB 00:00 2024-05-01T12:51:02.410 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:02.410 INFO:teuthology.orchestra.run.smithi063.stdout:Total 225 kB/s | 85 kB 00:00 2024-05-01T12:51:02.411 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:51:02.418 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:51:02.418 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:51:02.485 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:51:02.485 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:51:02.630 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:51:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:02.733 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:02.822 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-01T12:51:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-01T12:51:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:03.289 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:03.289 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:03.289 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:51:03.290 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:03.290 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:03.290 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:51:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-01T12:51:03.508 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-01T12:51:03.546 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-05-01T12:51:03.577 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-01T12:51:03.605 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 6/6 2024-05-01T12:51:04.081 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:54 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:51:04.205 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.1 M 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.1 M 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 13 M 2024-05-01T12:51:04.206 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 6/6 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 1/6 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-01T12:51:04.335 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-01T12:51:04.585 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-01T12:51:04.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:04.586 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:04.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-05-01T12:51:04.862 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-19.0.0-3122.gabc5b100.el9.x86_64.rpm 4.7 MB/s | 3.1 MB 00:00 2024-05-01T12:51:04.863 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:04.863 INFO:teuthology.orchestra.run.smithi063.stdout:Total 4.7 MB/s | 3.1 MB 00:00 2024-05-01T12:51:04.863 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:51:04.874 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:51:04.874 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:51:04.959 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:51:04.959 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:51:05.358 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:53 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:05.414 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:51:05.471 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 890 k 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-05-01T12:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 970 k 2024-05-01T12:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.8 M 2024-05-01T12:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:05.497 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:05.497 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:51:05.497 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-01T12:51:05.497 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-01T12:51:05.498 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-01T12:51:05.498 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:05.893 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-fuse-19.0.0-3122.gabc5b100.el9.x86_ 2.4 MB/s | 890 kB 00:00 2024-05-01T12:51:05.993 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 173 kB/s | 80 kB 00:00 2024-05-01T12:51:05.993 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:05.994 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.9 MB/s | 970 kB 00:00 2024-05-01T12:51:06.006 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:06.031 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:06.031 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:06.114 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:06.114 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:06.300 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:06.458 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-01T12:51:06.526 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2/2 2024-05-01T12:51:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:06.605 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:51:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:06.605 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:51:06.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-05-01T12:51:07.257 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2/2 2024-05-01T12:51:07.257 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/2 2024-05-01T12:51:07.390 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:57 ago on Wed 01 May 2024 12:49:10 PM UTC. 2024-05-01T12:51:07.502 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-01T12:51:07.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:07.503 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:07.503 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-01T12:51:07.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:07.503 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:07.516 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 170 k 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-05-01T12:51:07.517 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:07.518 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 170 k 2024-05-01T12:51:07.518 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 494 k 2024-05-01T12:51:07.518 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-05-01T12:51:07.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-05-01T12:51:07.849 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-19.0.0-3122.gabc5b100.el9.x86_64.rpm 516 kB/s | 170 kB 00:00 2024-05-01T12:51:07.849 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:07.849 INFO:teuthology.orchestra.run.smithi063.stdout:Total 513 kB/s | 170 kB 00:00 2024-05-01T12:51:07.850 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-05-01T12:51:07.858 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-05-01T12:51:07.858 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-05-01T12:51:07.925 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-05-01T12:51:07.925 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-05-01T12:51:08.084 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-05-01T12:51:08.182 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:08.306 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:56 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:08.435 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:08.435 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:08.435 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:51:08.435 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.0.0-3122.gabc5b100.el9 ceph-noarch 273 k 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-05-01T12:51:08.436 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:08.437 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 350 k 2024-05-01T12:51:08.437 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.5 M 2024-05-01T12:51:08.437 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:08.559 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:08.824 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 690 kB/s | 77 kB 00:00 2024-05-01T12:51:08.827 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:08.827 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:08.827 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-05-01T12:51:08.827 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:08.828 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:08.828 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-05-01T12:51:08.991 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-volume-19.0.0-3122.gabc5b100.el9.no 980 kB/s | 273 kB 00:00 2024-05-01T12:51:08.991 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:08.991 INFO:teuthology.orchestra.run.smithi050.stdout:Total 631 kB/s | 350 kB 00:00 2024-05-01T12:51:09.003 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:09.013 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:09.013 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:09.041 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:09.042 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:09.052 DEBUG:teuthology.parallel:result is None 2024-05-01T12:51:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-01T12:51:09.418 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2/2 2024-05-01T12:51:09.434 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2/2 2024-05-01T12:51:09.434 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:51:09.434 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-01T12:51:09.434 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:10.127 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 1/2 2024-05-01T12:51:10.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-01T12:51:10.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:10.444 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:10.445 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.0.0-3122.gabc5b100.el9.noarch 2024-05-01T12:51:10.445 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-01T12:51:10.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:10.445 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:10.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-05-01T12:51:11.265 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:59 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:11.388 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:11.389 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:11.389 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 127 k 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:11.390 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:51:11.391 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:11.391 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2024-05-01T12:51:11.391 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-05-01T12:51:11.391 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:11.614 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-19.0.0-3122.gabc5b100.el9.x86_64 565 kB/s | 127 kB 00:00 2024-05-01T12:51:11.615 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:11.615 INFO:teuthology.orchestra.run.smithi050.stdout:Total 561 kB/s | 127 kB 00:00 2024-05-01T12:51:11.615 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:11.621 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:11.621 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:11.689 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:11.689 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:12.635 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:12.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-05-01T12:51:13.386 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:01 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:13.468 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:13.512 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:13.513 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:13.513 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:13.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-05-01T12:51:14.165 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:02 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-05-01T12:51:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 32 k 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 32 k 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 142 k 2024-05-01T12:51:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:14.559 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-19.0.0-3122.gabc5b100.el9.x86_6 118 kB/s | 32 kB 00:00 2024-05-01T12:51:14.559 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:14.560 INFO:teuthology.orchestra.run.smithi050.stdout:Total 118 kB/s | 32 kB 00:00 2024-05-01T12:51:14.560 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:14.563 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:14.564 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:14.566 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:14.566 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:14.652 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:14.774 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:15.212 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:15.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-05-01T12:51:16.149 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:04 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:16.230 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:16.275 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:16.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-05-01T12:51:16.876 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:04 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:16.957 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:17.000 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:17.001 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:17.002 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:17.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-05-01T12:51:17.610 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:05 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:17.690 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:17.733 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:17.734 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:17.734 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:17.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-05-01T12:51:18.344 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:06 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:18.426 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:18.471 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:18.472 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:18.472 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:18.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-05-01T12:51:19.081 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:07 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:19.162 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:19.206 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:19.208 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:19.208 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:19.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-05-01T12:51:19.819 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:07 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:19.901 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:19.0.0-3122.gabc5b100.el9.x86_64 is already installed. 2024-05-01T12:51:19.946 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:19.947 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-05-01T12:51:19.947 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:20.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-05-01T12:51:20.553 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:08 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:20.677 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 85 k 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 85 k 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 231 k 2024-05-01T12:51:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:20.896 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-19.0.0-3122.gabc5b100.el9.x86_64.rpm 393 kB/s | 85 kB 00:00 2024-05-01T12:51:20.896 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:20.897 INFO:teuthology.orchestra.run.smithi050.stdout:Total 390 kB/s | 85 kB 00:00 2024-05-01T12:51:20.897 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:20.904 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:20.904 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:20.970 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:20.971 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:21.118 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:21.228 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:21.576 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:21.824 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:21.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:21.825 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:21.825 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:21.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:21.825 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:22.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-05-01T12:51:22.617 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:10 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:22.744 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:22.744 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:22.744 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-05-01T12:51:22.744 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 3.1 M 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:22.745 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.1 M 2024-05-01T12:51:22.746 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 13 M 2024-05-01T12:51:22.746 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:23.174 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-19.0.0-3122.gabc5b100.el9.x86_64.rpm 7.2 MB/s | 3.1 MB 00:00 2024-05-01T12:51:23.175 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:23.175 INFO:teuthology.orchestra.run.smithi050.stdout:Total 7.2 MB/s | 3.1 MB 00:00 2024-05-01T12:51:23.175 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:23.186 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:23.186 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:23.271 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:23.272 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:23.718 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-01T12:51:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:24.839 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:25.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-05-01T12:51:25.658 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:13 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:51:25.785 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:51:25.785 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.0.0-3122.gabc5b100.el9 ceph 170 k 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 170 k 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 494 k 2024-05-01T12:51:25.786 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:51:26.112 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-19.0.0-3122.gabc5b100.el9.x86_64.rpm 524 kB/s | 170 kB 00:00 2024-05-01T12:51:26.112 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:51:26.112 INFO:teuthology.orchestra.run.smithi050.stdout:Total 522 kB/s | 170 kB 00:00 2024-05-01T12:51:26.113 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:51:26.121 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:51:26.121 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:51:26.188 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:51:26.189 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:51:26.345 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:51:26.452 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:26.839 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:27.083 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 1/1 2024-05-01T12:51:27.084 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:27.084 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:51:27.084 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.0.0-3122.gabc5b100.el9.x86_64 2024-05-01T12:51:27.084 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:27.084 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:51:27.317 DEBUG:teuthology.parallel:result is None 2024-05-01T12:51:27.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=abc5b10088b379036a15386486ef8fee07d8cc3c 2024-05-01T12:51:27.546 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-01T12:51:27.567 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-3122.gabc5b100.el9 2024-05-01T12:51:27.568 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3122.gabc5b100.el9 2024-05-01T12:51:27.568 INFO:teuthology.task.install:The correct ceph version 19.0.0-3122.gabc5b100 is installed. 2024-05-01T12:51:27.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=abc5b10088b379036a15386486ef8fee07d8cc3c 2024-05-01T12:51:27.789 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-01T12:51:27.815 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-3122.gabc5b100.el9 2024-05-01T12:51:27.815 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3122.gabc5b100.el9 2024-05-01T12:51:27.815 INFO:teuthology.task.install:The correct ceph version 19.0.0-3122.gabc5b100 is installed. 2024-05-01T12:51:27.817 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-01T12:51:27.817 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:27.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-01T12:51:27.843 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:27.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-01T12:51:27.886 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-01T12:51:27.886 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:27.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-05-01T12:51:27.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-01T12:51:27.974 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:27.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-05-01T12:51:28.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-01T12:51:28.071 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-01T12:51:28.071 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:28.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-01T12:51:28.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-01T12:51:28.160 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:28.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-01T12:51:28.184 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-01T12:51:28.249 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-01T12:51:28.249 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:28.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-05-01T12:51:28.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-01T12:51:28.337 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:28.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-05-01T12:51:28.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-01T12:51:28.428 INFO:teuthology.run_tasks:Running task ceph... 2024-05-01T12:51:28.518 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-01T12:51:28.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-05-01T12:51:28.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-05-01T12:51:28.544 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-01T12:51:28.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-01T12:51:28.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-01T12:51:28.609 INFO:tasks.ceph:Creating extra log directories... 2024-05-01T12:51:28.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-01T12:51:28.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-01T12:51:28.685 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-01T12:51:28.686 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-01T12:51:28.686 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi/7682921', 'branch': 'wip-alimasa-bn-test', 'description': 'rgw:notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/others/{0-install supported-distros/{centos_latest} test_others}}', 'email': None, 'first_in_suite': False, 'job_id': '7682921', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'alimasa-2024-05-01_12:37:01-rgw:notifications-wip-alimasa-bn-test-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-alimasa-bn-test'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c'}}, 'rgw': {'frontend': 'beast', 'realm': 'MyRealm', 'storage classes': 'LUKEWARM, FROZEN', 'zone': 'MyZone', 'zonegroup': 'MyZoneGroup'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-alimasa-bn-test-testing', 'sha1': '971a59c7a133e90e9bfb891a1b11587cefa2ea03'}}, 'owner': 'scheduled_alimasa@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': 'abc5b10088b379036a15386486ef8fee07d8cc3c', 'sleep_before_teardown': 0, 'suite': 'rgw:notifications', 'suite_branch': 'wip-alimasa-bn-test-testing', 'suite_path': '/home/teuthworker/src/github.com_AliMasarweh_ceph_971a59c7a133e90e9bfb891a1b11587cefa2ea03/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/AliMasarweh/ceph.git', 'suite_sha1': '971a59c7a133e90e9bfb891a1b11587cefa2ea03', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlyqt9Hkbw7aMOo6S+f2Sg5D459tcj8ks9Myzxf7bJEJAVb0kTxRr/H1C7BT/8wL/anPuInHWl0ssxj2FpX9Y=', 'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLG4sVglRCzJsU4+TxHf4i78tWMLMAmZKIVu1h9DesveLneCAUiVtS/k9oSLU85IJbAP8zr+7tG7CDeebi09xQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None, 'client.1': None}}, {'notification-tests': {'client.0': {'rgw_server': 'client.0'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-05-01_12:37:01', 'tube': 'smithi', 'user': 'alimasa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-01T12:51:28.686 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-01T12:51:28.714 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-01T12:51:28.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-05-01T12:51:28.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-05-01T12:51:28.794 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:28.794 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-05-01T12:51:28.847 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-01T12:51:28.847 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-05-01T12:51:28.902 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-01 12:51:24.050483328 +0000 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-01 12:48:29.349493202 +0000 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-01 12:48:29.349493202 +0000 2024-05-01T12:51:28.903 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-01 12:48:29.349493202 +0000 2024-05-01T12:51:28.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-01T12:51:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-01T12:51:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-01T12:51:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000100927 s, 5.1 MB/s 2024-05-01T12:51:28.967 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-01T12:51:29.024 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-05-01T12:51:29.079 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-01 12:51:24.051483302 +0000 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-01 12:48:29.683484264 +0000 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-01 12:48:29.683484264 +0000 2024-05-01T12:51:29.080 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-01 12:48:29.683484264 +0000 2024-05-01T12:51:29.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-01T12:51:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-01T12:51:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-01T12:51:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.8266e-05 s, 5.2 MB/s 2024-05-01T12:51:29.144 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-01T12:51:29.200 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-05-01T12:51:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-01T12:51:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.255 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-01 12:51:24.051483302 +0000 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-01 12:48:30.020475246 +0000 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-01 12:48:30.020475246 +0000 2024-05-01T12:51:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-01 12:48:30.020475246 +0000 2024-05-01T12:51:29.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-01T12:51:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-01T12:51:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-01T12:51:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.8323e-05 s, 5.2 MB/s 2024-05-01T12:51:29.319 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-01T12:51:29.375 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-05-01T12:51:29.431 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-01T12:51:29.431 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.431 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-05-01 12:51:24.051483302 +0000 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-05-01 12:48:30.353466334 +0000 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-05-01 12:48:30.353466334 +0000 2024-05-01T12:51:29.432 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-05-01 12:48:30.353466334 +0000 2024-05-01T12:51:29.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-01T12:51:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-05-01T12:51:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-05-01T12:51:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.5337e-05 s, 5.4 MB/s 2024-05-01T12:51:29.495 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-01T12:51:29.551 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:29.551 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:29.551 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-05-01T12:51:29.568 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-01T12:51:29.568 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-05-01 12:51:05.750422745 +0000 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-05-01 12:48:29.361244279 +0000 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-05-01 12:48:29.361244279 +0000 2024-05-01T12:51:29.624 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-05-01 12:48:29.361244279 +0000 2024-05-01T12:51:29.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-01T12:51:29.687 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-05-01T12:51:29.687 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-05-01T12:51:29.687 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000207299 s, 2.5 MB/s 2024-05-01T12:51:29.688 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-01T12:51:29.744 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-05-01T12:51:29.799 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-01T12:51:29.799 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-05-01 12:51:05.750422745 +0000 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-05-01 12:48:29.707240777 +0000 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-05-01 12:48:29.707240777 +0000 2024-05-01T12:51:29.800 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-05-01 12:48:29.707240777 +0000 2024-05-01T12:51:29.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-01T12:51:29.861 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-05-01T12:51:29.862 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-05-01T12:51:29.862 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000206317 s, 2.5 MB/s 2024-05-01T12:51:29.863 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-01T12:51:29.919 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-05-01T12:51:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-01T12:51:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:29.974 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-05-01 12:51:05.750422745 +0000 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-05-01 12:48:30.056237244 +0000 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-05-01 12:48:30.056237244 +0000 2024-05-01T12:51:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-05-01 12:48:30.056237244 +0000 2024-05-01T12:51:29.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-01T12:51:30.037 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-05-01T12:51:30.037 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-05-01T12:51:30.037 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000162735 s, 3.1 MB/s 2024-05-01T12:51:30.038 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-01T12:51:30.094 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-05-01 12:51:05.751422735 +0000 2024-05-01T12:51:30.149 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-05-01 12:48:30.402233742 +0000 2024-05-01T12:51:30.150 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-05-01 12:48:30.402233742 +0000 2024-05-01T12:51:30.150 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-05-01 12:48:30.402233742 +0000 2024-05-01T12:51:30.150 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-01T12:51:30.212 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-05-01T12:51:30.212 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-05-01T12:51:30.212 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000322651 s, 1.6 MB/s 2024-05-01T12:51:30.213 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-01T12:51:30.269 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:30.269 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-01T12:51:30.269 INFO:tasks.ceph:Generating config... 2024-05-01T12:51:30.270 INFO:tasks.ceph:[client] debug rgw = 20 2024-05-01T12:51:30.270 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-05-01T12:51:30.270 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-05-01T12:51:30.270 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-05-01T12:51:30.271 INFO:tasks.ceph:[client] setgroup = ceph 2024-05-01T12:51:30.271 INFO:tasks.ceph:[client] setuser = ceph 2024-05-01T12:51:30.271 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2024-05-01T12:51:30.271 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2024-05-01T12:51:30.271 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-01T12:51:30.271 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-01T12:51:30.271 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-01T12:51:30.271 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-01T12:51:30.271 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-01T12:51:30.271 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-01T12:51:30.272 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-01T12:51:30.272 INFO:tasks.ceph:Setting up mon.a... 2024-05-01T12:51:30.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-01T12:51:30.325 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2024-05-01T12:51:30.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-01T12:51:30.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-01T12:51:30.436 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50'), ('mon.c', '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]'), ('mon.b', '172.21.15.63')] 2024-05-01T12:51:30.436 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.63', 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-01T12:51:30.437 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:30.437 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-01T12:51:30.491 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --addv c '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]' --add b 172.21.15.63 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:30.577 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:51:30.577 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid fcc845ba-0166-4b8a-a069-0a6a9d719b5c 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:fsid fcc845ba-0166-4b8a-a069-0a6a9d719b5c 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2024-05-01T12:51:30.578331+0000 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-05-01T12:51:30.578331+0000 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 17 (quincy) 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-05-01T12:51:30.578 INFO:teuthology.orchestra.run.smithi050.stdout:1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.b 2024-05-01T12:51:30.579 INFO:teuthology.orchestra.run.smithi050.stdout:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-05-01T12:51:30.579 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-01T12:51:30.580 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-01T12:51:30.594 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fcc845ba-0166-4b8a-a069-0a6a9d719b5c... 2024-05-01T12:51:30.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-01T12:51:30.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-01T12:51:30.675 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-05-01T12:51:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-05-01T12:51:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-05-01T12:51:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-05-01T12:51:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.63" 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_pg_log_entries = 10 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd_min_pg_log_entries = 10 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = fcc845ba-0166-4b8a-a069-0a6a9d719b5c 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug trim objects = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-05-01T12:51:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore allocator = bitmap 2024-05-01T12:51:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 1/20 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 1/20 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 4/10 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.683 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-05-01T12:51:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-05-01T12:51:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: debug rgw = 20 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt require ssl = False 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms backend = testing 2024-05-01T12:51:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: setgroup = ceph 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: setuser = ceph 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-05-01T12:51:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-01T12:51:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-01T12:51:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = "172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.63" 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_pg_log_entries = 10 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: osd_min_pg_log_entries = 10 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = fcc845ba-0166-4b8a-a069-0a6a9d719b5c 2024-05-01T12:51:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: bdev async discard = True 2024-05-01T12:51:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: bdev enable discard = True 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore allocator = bitmap 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore block size = 96636764160 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore fsck on mount = True 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 1/20 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 1/20 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 4/10 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd backfillfull_ratio = 0.85 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd full ratio = 0.9 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd nearfull ratio = 0.8 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: osd failsafe full ratio = 0.95 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = bluestore 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: osd shutdown pgref assert = True 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-05-01T12:51:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-05-01T12:51:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-05-01T12:51:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 20 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt require ssl = False 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms backend = testing 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-05-01T12:51:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: setgroup = ceph 2024-05-01T12:51:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: setuser = ceph 2024-05-01T12:51:30.697 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-05-01T12:51:30.697 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.c] 2024-05-01T12:51:30.697 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.b] 2024-05-01T12:51:30.698 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-01T12:51:30.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-01T12:51:30.791 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-01T12:51:30.791 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:30.791 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-01T12:51:30.806 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:30.806 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-01T12:51:30.861 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:51:30.861 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:30.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-01T12:51:30.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-01T12:51:30.935 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:30.935 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:30.990 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:51:30.990 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:30.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-01T12:51:30.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-01T12:51:31.028 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:31.028 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:31.084 INFO:tasks.ceph:Setting up mon nodes... 2024-05-01T12:51:31.085 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-01T12:51:31.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-01T12:51:31.134 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-01T12:51:31.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-01T12:51:31.197 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-01T12:51:31.200 INFO:tasks.ceph:Setting up mds nodes... 2024-05-01T12:51:31.200 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-01T12:51:31.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-01T12:51:31.240 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-01T12:51:31.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-01T12:51:31.293 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-01T12:51:31.306 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-01T12:51:31.306 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-01T12:51:31.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-01T12:51:31.330 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:31.330 INFO:tasks.ceph:role: osd.0 2024-05-01T12:51:31.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:51:31.330 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-01T12:51:31.394 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:31.394 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:31.395 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:31.395 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:31.395 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:31.427 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-05-01T12:51:31.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-05-01T12:51:31.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-01T12:51:31.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-01T12:51:31.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-01T12:51:31.631 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:31.631 INFO:tasks.ceph:role: osd.1 2024-05-01T12:51:31.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:51:31.632 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:31.727 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-05-01T12:51:31.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-05-01T12:51:31.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-01T12:51:31.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-01T12:51:31.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-01T12:51:31.930 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:31.930 INFO:tasks.ceph:role: osd.2 2024-05-01T12:51:31.930 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:51:31.930 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-01T12:51:31.994 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:31.994 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:31.994 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:31.994 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:31.994 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:31.995 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:31.995 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:31.995 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:31.995 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:31.995 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:32.027 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-05-01T12:51:32.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-05-01T12:51:32.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-01T12:51:32.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-01T12:51:32.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-01T12:51:32.229 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:32.229 INFO:tasks.ceph:role: osd.3 2024-05-01T12:51:32.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2024-05-01T12:51:32.230 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:32.293 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:32.294 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:32.294 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:32.325 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-05-01T12:51:32.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-05-01T12:51:32.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-01T12:51:32.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-01T12:51:32.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:32.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.553+0000 7fdfcd01c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-01T12:51:32.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.553+0000 7fdfcd01c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-01T12:51:32.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.553+0000 7fdfcd01c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-01T12:51:32.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.554+0000 7fdfcd01c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-01T12:51:32.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.554+0000 7fdfcd01c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-01T12:51:32.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:32.554+0000 7fdfcd01c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-01T12:51:35.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-01T12:51:35.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.523+0000 7fdfa661f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-01T12:51:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.524+0000 7fdfa661f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-01T12:51:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.524+0000 7fdfa661f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-01T12:51:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.524+0000 7fdfa661f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-01T12:51:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.524+0000 7fdfa661f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-01T12:51:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:35.524+0000 7fdfa661f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-01T12:51:38.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-01T12:51:38.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.493+0000 7f57fb744640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.493+0000 7f57fb744640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.493+0000 7f57fb744640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.493+0000 7f57fb744640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.493+0000 7f57fb744640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-01T12:51:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:38.494+0000 7f57fb744640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-01T12:51:41.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-01T12:51:41.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.472+0000 7f39d7e10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-01T12:51:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.473+0000 7f39d7e10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-01T12:51:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.473+0000 7f39d7e10640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-01T12:51:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.473+0000 7f39d7e10640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-01T12:51:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.473+0000 7f39d7e10640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-01T12:51:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:51:41.473+0000 7f39d7e10640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-01T12:51:44.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-01T12:51:44.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-01T12:51:44.384 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:44.384 INFO:tasks.ceph:role: osd.4 2024-05-01T12:51:44.384 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:51:44.385 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-01T12:51:44.459 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:44.460 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:44.969 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-05-01T12:51:44.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-05-01T12:51:44.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-01T12:51:45.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-01T12:51:45.118 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-01T12:51:45.185 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:45.185 INFO:tasks.ceph:role: osd.5 2024-05-01T12:51:45.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:51:45.185 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-01T12:51:45.258 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:45.259 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:45.718 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-05-01T12:51:45.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-05-01T12:51:45.721 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-01T12:51:45.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-01T12:51:45.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-01T12:51:45.908 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:45.908 INFO:tasks.ceph:role: osd.6 2024-05-01T12:51:45.908 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:51:45.908 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-01T12:51:45.979 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:45.979 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:45.979 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:45.980 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:46.472 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-05-01T12:51:46.474 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-05-01T12:51:46.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-01T12:51:46.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-01T12:51:46.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-01T12:51:46.715 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-01T12:51:46.716 INFO:tasks.ceph:role: osd.7 2024-05-01T12:51:46.716 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com 2024-05-01T12:51:46.716 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-01T12:51:46.788 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-01T12:51:46.789 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-01T12:51:46.789 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-01T12:51:47.291 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-05-01T12:51:47.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-05-01T12:51:47.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-01T12:51:47.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-01T12:51:47.480 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.580+0000 7fa53b133640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.581+0000 7fa53b133640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.581+0000 7fa53b133640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.581+0000 7fa53b133640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.581+0000 7fa53b133640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-01T12:51:47.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:47.581+0000 7fa53b133640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-01T12:51:50.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-01T12:51:50.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:50.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.553+0000 7f5e02423640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-01T12:51:50.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.554+0000 7f5e02423640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-01T12:51:50.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.554+0000 7f5e02423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-01T12:51:50.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.554+0000 7f5e02423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-01T12:51:50.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.554+0000 7f5e02423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-01T12:51:50.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:50.554+0000 7f5e02423640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-01T12:51:53.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-01T12:51:53.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:53.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.532+0000 7f08f6ca0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-01T12:51:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.532+0000 7f08f6ca0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-01T12:51:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.532+0000 7f08f6ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-01T12:51:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.532+0000 7f08f6ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-01T12:51:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.532+0000 7f08f6ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-01T12:51:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:53.533+0000 7f08f6ca0640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-01T12:51:56.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-01T12:51:56.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:51:56.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.502+0000 7ffa8f610640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-01T12:51:56.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.502+0000 7ffa8f610640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-01T12:51:56.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.502+0000 7ffa8f610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-01T12:51:56.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.503+0000 7ffa8f610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-01T12:51:56.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.503+0000 7ffa8f610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-01T12:51:56.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:51:56.503+0000 7ffa8f610640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-01T12:51:59.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-01T12:51:59.383 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-01T12:51:59.384 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-01T12:51:59.414 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-01T12:51:59.483 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-01T12:51:59.551 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-01T12:51:59.619 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-01T12:51:59.686 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:59.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-01T12:51:59.716 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:59.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-01T12:51:59.785 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:59.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-01T12:51:59.853 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:59.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-01T12:51:59.920 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:51:59.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-01T12:51:59.986 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:51:59.986 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-01T12:52:00.005 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:52:00.005 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-01T12:52:00.042 INFO:tasks.ceph:Adding keys to all mons... 2024-05-01T12:52:00.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-01T12:52:00.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDTOjJmCWMPCBAAXZu0pmZSZqcOjX2ZIT3x5Q== 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDUOjJmRlIRIRAANXqHy8NxsNKK/N9UwIv+eQ== 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDXOjJmosdIHxAAxq/ZUwVHp468mkvBAXpOUw== 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDaOjJmpqB3HRAAUiFzhZsQJKIlHVh3tf/H0g== 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDdOjJm7m49HBAA5FmElnGZqkp78j+3YgqQsg== 2024-05-01T12:52:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDTOjJm0U/UCxAA/C0IpXziDKyJalZSA+tZ8Q== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDjOjJmRyW/IhAAlDPYaDz2sdksKeoOC+wuhg== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDmOjJmSuAfIRAAh3ELZtcU3iwFOCV9ZV2RuA== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDpOjJmyAbHHxAASqBkUO64OB9SrX8VR+uBsA== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDsOjJmXr4EHhAAai7H5HnAOTvJDmxKHO7piA== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDTOjJmcPJmDhAAlWQ0VzaA+tqG12x6vUICVw== 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2024-05-01T12:52:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDTOjJm6MiNERAAiD9jhiaLlkF1CWG3zodLzQ== 2024-05-01T12:52:00.106 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.y] 2024-05-01T12:52:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDTOjJmCWMPCBAAXZu0pmZSZqcOjX2ZIT3x5Q== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDUOjJmRlIRIRAANXqHy8NxsNKK/N9UwIv+eQ== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDXOjJmosdIHxAAxq/ZUwVHp468mkvBAXpOUw== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDaOjJmpqB3HRAAUiFzhZsQJKIlHVh3tf/H0g== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDdOjJm7m49HBAA5FmElnGZqkp78j+3YgqQsg== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDTOjJm0U/UCxAA/C0IpXziDKyJalZSA+tZ8Q== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.4] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDjOjJmRyW/IhAAlDPYaDz2sdksKeoOC+wuhg== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.5] 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDmOjJmSuAfIRAAh3ELZtcU3iwFOCV9ZV2RuA== 2024-05-01T12:52:00.107 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.6] 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDpOjJmyAbHHxAASqBkUO64OB9SrX8VR+uBsA== 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.7] 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDsOjJmXr4EHhAAai7H5HnAOTvJDmxKHO7piA== 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDTOjJmcPJmDhAAlWQ0VzaA+tqG12x6vUICVw== 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2024-05-01T12:52:00.108 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDTOjJm6MiNERAAiD9jhiaLlkF1CWG3zodLzQ== 2024-05-01T12:52:00.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-01T12:52:00.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-01T12:52:00.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.204 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-01T12:52:00.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-01T12:52:00.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.569 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.621 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-01T12:52:00.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-01T12:52:00.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-01T12:52:00.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-01T12:52:00.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-01T12:52:00.780 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-01T12:52:00.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-01T12:52:00.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-01T12:52:00.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-01T12:52:00.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-01T12:52:00.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-01T12:52:01.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-01T12:52:01.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-01T12:52:01.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-01T12:52:01.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-01T12:52:01.253 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:52:01.256 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-01T12:52:01.309 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-01T12:52:01.309 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-01T12:52:01.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-01T12:52:01.311 INFO:tasks.ceph.mon.a:Started 2024-05-01T12:52:01.312 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-01T12:52:01.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-01T12:52:01.313 INFO:tasks.ceph.mon.c:Started 2024-05-01T12:52:01.314 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-01T12:52:01.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-01T12:52:01.352 INFO:tasks.ceph.mon.b:Started 2024-05-01T12:52:01.352 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-01T12:52:01.352 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-01T12:52:01.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-01T12:52:01.354 INFO:tasks.ceph.mgr.y:Started 2024-05-01T12:52:01.355 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-01T12:52:01.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-01T12:52:01.357 INFO:tasks.ceph.mgr.x:Started 2024-05-01T12:52:01.357 INFO:tasks.ceph:Setting crush tunables to default 2024-05-01T12:52:01.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2024-05-01T12:52:01.695 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-05-01T12:52:01.707 INFO:tasks.ceph:check_enable_crimson: False 2024-05-01T12:52:01.707 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-01T12:52:01.707 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:52:01.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-01T12:52:01.726 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:01.726+0000 7f4a0ea41040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-01T12:52:01.731 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:52:01.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-01T12:52:01.734 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:01.734+0000 7ffab5649040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-01T12:52:01.758 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:52:01.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-01T12:52:01.823 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:52:01.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-01T12:52:01.880 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:01.879+0000 7f4a0ea41040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-01T12:52:01.888 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:52:01.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-01T12:52:01.912 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:52:01.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-01T12:52:01.918 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:01.917+0000 7ffab5649040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-01T12:52:01.936 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:01.936+0000 7f4a0ea41040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-01T12:52:01.960 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:52:01.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-01T12:52:01.973 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:01.973+0000 7ffab5649040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-01T12:52:02.024 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-05-01T12:52:02.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-01T12:52:02.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 5bfbb253-e04e-42e1-af0a-c24d03aee610 0 2024-05-01T12:52:02.191 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:02.191+0000 7f4a0ea41040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-01T12:52:02.233 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:02.233+0000 7ffab5649040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-01T12:52:02.244 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:02.243+0000 7f4a0ea41040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-01T12:52:02.281 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:02.281+0000 7ffab5649040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-01T12:52:02.329 INFO:teuthology.orchestra.run.smithi063.stdout:0 2024-05-01T12:52:02.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 85ce1d73-dcc0-43ca-ba94-e681b20d7053 1 2024-05-01T12:52:02.529 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-05-01T12:52:02.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new d7dac084-480e-4f7b-a6bf-194e27188ab1 2 2024-05-01T12:52:02.729 INFO:teuthology.orchestra.run.smithi063.stdout:2 2024-05-01T12:52:02.740 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 96dd3a06-bcc8-4d86-8720-91a3d98b3558 3 2024-05-01T12:52:02.927 INFO:teuthology.orchestra.run.smithi063.stdout:3 2024-05-01T12:52:02.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new ec3a7c20-0e93-4431-96d7-24f75bc43093 4 2024-05-01T12:52:02.989 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:02.989+0000 7f4a0ea41040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-01T12:52:03.004 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.004+0000 7ffab5649040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-01T12:52:03.092 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.092+0000 7f4a0ea41040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-01T12:52:03.096 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.096+0000 7ffab5649040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-01T12:52:03.137 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.137+0000 7f4a0ea41040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-01T12:52:03.137 INFO:teuthology.orchestra.run.smithi063.stdout:4 2024-05-01T12:52:03.141 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.140+0000 7ffab5649040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-01T12:52:03.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new a5420e0a-8bc2-4217-946c-1b4a3baa3e36 5 2024-05-01T12:52:03.314 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.313+0000 7ffab5649040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-01T12:52:03.316 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.315+0000 7f4a0ea41040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-01T12:52:03.332 INFO:teuthology.orchestra.run.smithi063.stdout:5 2024-05-01T12:52:03.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new db15cd66-2173-4e9a-9197-270c8ede9194 6 2024-05-01T12:52:03.363 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.363+0000 7ffab5649040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-01T12:52:03.376 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.376+0000 7f4a0ea41040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-01T12:52:03.419 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.419+0000 7ffab5649040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-01T12:52:03.433 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.433+0000 7f4a0ea41040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-01T12:52:03.527 INFO:teuthology.orchestra.run.smithi063.stdout:6 2024-05-01T12:52:03.537 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.537+0000 7ffab5649040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-01T12:52:03.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 8b13f614-d1da-4b89-b91f-e530ce24febf 7 2024-05-01T12:52:03.554 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.553+0000 7f4a0ea41040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-01T12:52:03.739 INFO:teuthology.orchestra.run.smithi063.stdout:7 2024-05-01T12:52:03.751 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-01T12:52:03.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-01T12:52:03.753 INFO:tasks.ceph.osd.0:Started 2024-05-01T12:52:03.753 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-01T12:52:03.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-01T12:52:03.755 INFO:tasks.ceph.osd.1:Started 2024-05-01T12:52:03.756 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-01T12:52:03.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-01T12:52:03.758 INFO:tasks.ceph.osd.2:Started 2024-05-01T12:52:03.758 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-01T12:52:03.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-01T12:52:03.760 INFO:tasks.ceph.osd.3:Started 2024-05-01T12:52:03.760 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-01T12:52:03.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-01T12:52:03.762 INFO:tasks.ceph.osd.4:Started 2024-05-01T12:52:03.762 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-01T12:52:03.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-01T12:52:03.764 INFO:tasks.ceph.osd.5:Started 2024-05-01T12:52:03.765 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-01T12:52:03.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-01T12:52:03.766 INFO:tasks.ceph.osd.6:Started 2024-05-01T12:52:03.767 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-01T12:52:03.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-01T12:52:03.769 INFO:tasks.ceph.osd.7:Started 2024-05-01T12:52:03.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-01T12:52:03.968 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:03.968+0000 7ffab5649040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-01T12:52:03.969 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:03.969 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"fsid":"fcc845ba-0166-4b8a-a069-0a6a9d719b5c","created":"2024-05-01T12:52:01.606905+0000","modified":"2024-05-01T12:52:03.737913+0000","last_up_change":"0.000000","last_in_change":"2024-05-01T12:52:03.737913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5bfbb253-e04e-42e1-af0a-c24d03aee610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85ce1d73-dcc0-43ca-ba94-e681b20d7053","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7dac084-480e-4f7b-a6bf-194e27188ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"96dd3a06-bcc8-4d86-8720-91a3d98b3558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ec3a7c20-0e93-4431-96d7-24f75bc43093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a5420e0a-8bc2-4217-946c-1b4a3baa3e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"db15cd66-2173-4e9a-9197-270c8ede9194","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8b13f614-d1da-4b89-b91f-e530ce24febf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-01T12:52:03.980 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-01T12:52:03.980 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-01T12:52:03.995 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:03.995+0000 7f4a0ea41040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-01T12:52:04.239 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.239+0000 7ffab5649040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-01T12:52:04.270 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.267+0000 7f4a0ea41040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-01T12:52:04.293 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.293+0000 7ffab5649040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-01T12:52:04.322 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.322+0000 7f4a0ea41040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-01T12:52:04.363 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T12:52:04.362+0000 7fe6a2c12640 -1 Falling back to public interface 2024-05-01T12:52:04.365 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T12:52:04.364+0000 7f1dba51a640 -1 Falling back to public interface 2024-05-01T12:52:04.366 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T12:52:04.365+0000 7fe1dd80f640 -1 Falling back to public interface 2024-05-01T12:52:04.366 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T12:52:04.365+0000 7fb422e0e640 -1 Falling back to public interface 2024-05-01T12:52:04.376 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T12:52:04.375+0000 7f07d7c2c640 -1 Falling back to public interface 2024-05-01T12:52:04.380 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T12:52:04.379+0000 7f9b6b2ac640 -1 Falling back to public interface 2024-05-01T12:52:04.384 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T12:52:04.383+0000 7f8f7f61f640 -1 Falling back to public interface 2024-05-01T12:52:04.385 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T12:52:04.384+0000 7f245d2a0640 -1 Falling back to public interface 2024-05-01T12:52:04.419 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.419+0000 7ffab5649040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-01T12:52:04.447 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.447+0000 7f4a0ea41040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-01T12:52:04.712 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.711+0000 7ffab5649040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-01T12:52:04.738 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.738+0000 7f4a0ea41040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-01T12:52:04.761 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.761+0000 7ffab5649040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-01T12:52:04.786 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-01T12:52:04.788 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.787+0000 7f4a0ea41040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-01T12:52:04.929 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:04.928+0000 7ffab5649040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-01T12:52:04.951 INFO:teuthology.misc.health.smithi050.stdout: 2024-05-01T12:52:04.951 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"fcc845ba-0166-4b8a-a069-0a6a9d719b5c","created":"2024-05-01T12:52:01.606905+0000","modified":"2024-05-01T12:52:03.737913+0000","last_up_change":"0.000000","last_in_change":"2024-05-01T12:52:03.737913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5bfbb253-e04e-42e1-af0a-c24d03aee610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85ce1d73-dcc0-43ca-ba94-e681b20d7053","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7dac084-480e-4f7b-a6bf-194e27188ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"96dd3a06-bcc8-4d86-8720-91a3d98b3558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ec3a7c20-0e93-4431-96d7-24f75bc43093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a5420e0a-8bc2-4217-946c-1b4a3baa3e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"db15cd66-2173-4e9a-9197-270c8ede9194","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8b13f614-d1da-4b89-b91f-e530ce24febf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-01T12:52:04.952 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:04.951+0000 7f4a0ea41040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-01T12:52:04.961 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-01T12:52:05.072 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.072+0000 7ffab5649040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-01T12:52:05.095 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.095+0000 7f4a0ea41040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-01T12:52:05.124 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.124+0000 7ffab5649040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-01T12:52:05.147 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.147+0000 7f4a0ea41040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-01T12:52:05.197 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.197+0000 7ffab5649040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-01T12:52:05.221 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.220+0000 7f4a0ea41040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-01T12:52:05.245 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.245+0000 7ffab5649040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-01T12:52:05.268 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.268+0000 7f4a0ea41040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-01T12:52:05.744 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.743+0000 7ffab5649040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-01T12:52:05.767 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.767+0000 7f4a0ea41040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-01T12:52:05.865 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:05.864+0000 7ffab5649040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-01T12:52:05.889 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-05-01T12:52:05.888+0000 7f4a0ea41040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-01T12:52:06.879 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:06.878+0000 7ffa562fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-01T12:52:06.879 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-05-01T12:52:06.878+0000 7ffa562fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-01T12:52:07.825 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T12:52:07.824+0000 7fb422e0e640 -1 osd.5 0 log_to_monitors true 2024-05-01T12:52:07.866 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T12:52:07.866+0000 7fe6a2c12640 -1 osd.0 0 log_to_monitors true 2024-05-01T12:52:07.869 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T12:52:07.869+0000 7f9b6b2ac640 -1 osd.3 0 log_to_monitors true 2024-05-01T12:52:07.872 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T12:52:07.872+0000 7f07d7c2c640 -1 osd.2 0 log_to_monitors true 2024-05-01T12:52:07.884 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T12:52:07.884+0000 7f8f7f61f640 -1 osd.1 0 log_to_monitors true 2024-05-01T12:52:07.898 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T12:52:07.897+0000 7fe1dd80f640 -1 osd.4 0 log_to_monitors true 2024-05-01T12:52:07.906 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T12:52:07.906+0000 7f245d2a0640 -1 osd.6 0 log_to_monitors true 2024-05-01T12:52:07.910 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T12:52:07.909+0000 7f1dba51a640 -1 osd.7 0 log_to_monitors true 2024-05-01T12:52:09.636 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T12:52:09.635+0000 7fe69e651640 -1 osd.0 0 waiting for initial osdmap 2024-05-01T12:52:09.648 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T12:52:09.648+0000 7fe696471640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:09.796 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T12:52:09.796+0000 7fb41e754640 -1 osd.5 0 waiting for initial osdmap 2024-05-01T12:52:09.805 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T12:52:09.805+0000 7fb416574640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.115 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T12:52:11.115+0000 7f9b67242640 -1 osd.3 0 waiting for initial osdmap 2024-05-01T12:52:11.124 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T12:52:11.124+0000 7f9b5f062640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.129 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T12:52:11.129+0000 7f07d31c3640 -1 osd.2 0 waiting for initial osdmap 2024-05-01T12:52:11.141 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T12:52:11.141+0000 7f07cafe3640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.147 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T12:52:11.147+0000 7f8f7b77b640 -1 osd.1 0 waiting for initial osdmap 2024-05-01T12:52:11.157 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T12:52:11.157+0000 7f8f72d87640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.511 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T12:52:11.511+0000 7f1db64b0640 -1 osd.7 0 waiting for initial osdmap 2024-05-01T12:52:11.527 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T12:52:11.527+0000 7f1dae2d0640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.573 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T12:52:11.572+0000 7f2459236640 -1 osd.6 0 waiting for initial osdmap 2024-05-01T12:52:11.583 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T12:52:11.583+0000 7f2451056640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.678 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T12:52:11.678+0000 7fe1d9222640 -1 osd.4 0 waiting for initial osdmap 2024-05-01T12:52:11.692 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T12:52:11.692+0000 7fe1d1042640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-01T12:52:11.767 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-01T12:52:12.073 INFO:teuthology.misc.health.smithi050.stdout: 2024-05-01T12:52:12.073 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":15,"fsid":"fcc845ba-0166-4b8a-a069-0a6a9d719b5c","created":"2024-05-01T12:52:01.606905+0000","modified":"2024-05-01T12:52:11.893267+0000","last_up_change":"2024-05-01T12:52:11.893267+0000","last_in_change":"2024-05-01T12:52:03.737913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-01T12:52:09.893222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfbb253-e04e-42e1-af0a-c24d03aee610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6801","nonce":2305361892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6803","nonce":2305361892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6807","nonce":2305361892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6805","nonce":2305361892}]},"public_addr":"172.21.15.50:6801/2305361892","cluster_addr":"172.21.15.50:6803/2305361892","heartbeat_back_addr":"172.21.15.50:6807/2305361892","heartbeat_front_addr":"172.21.15.50:6805/2305361892","state":["exists","up"]},{"osd":1,"uuid":"85ce1d73-dcc0-43ca-ba94-e681b20d7053","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6825","nonce":247574176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6827","nonce":247574176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6831","nonce":247574176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6829","nonce":247574176}]},"public_addr":"172.21.15.50:6825/247574176","cluster_addr":"172.21.15.50:6827/247574176","heartbeat_back_addr":"172.21.15.50:6831/247574176","heartbeat_front_addr":"172.21.15.50:6829/247574176","state":["exists","up"]},{"osd":2,"uuid":"d7dac084-480e-4f7b-a6bf-194e27188ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6809","nonce":1822591763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6811","nonce":1822591763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6815","nonce":1822591763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6813","nonce":1822591763}]},"public_addr":"172.21.15.50:6809/1822591763","cluster_addr":"172.21.15.50:6811/1822591763","heartbeat_back_addr":"172.21.15.50:6815/1822591763","heartbeat_front_addr":"172.21.15.50:6813/1822591763","state":["exists","up"]},{"osd":3,"uuid":"96dd3a06-bcc8-4d86-8720-91a3d98b3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6817","nonce":2001127933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6819","nonce":2001127933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6823","nonce":2001127933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6821","nonce":2001127933}]},"public_addr":"172.21.15.50:6817/2001127933","cluster_addr":"172.21.15.50:6819/2001127933","heartbeat_back_addr":"172.21.15.50:6823/2001127933","heartbeat_front_addr":"172.21.15.50:6821/2001127933","state":["exists","up"]},{"osd":4,"uuid":"ec3a7c20-0e93-4431-96d7-24f75bc43093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6811","nonce":2421506900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6817","nonce":2421506900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6823","nonce":2421506900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6821","nonce":2421506900}]},"public_addr":"172.21.15.63:6811/2421506900","cluster_addr":"172.21.15.63:6817/2421506900","heartbeat_back_addr":"172.21.15.63:6823/2421506900","heartbeat_front_addr":"172.21.15.63:6821/2421506900","state":["exists","up"]},{"osd":5,"uuid":"a5420e0a-8bc2-4217-946c-1b4a3baa3e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6810","nonce":173493555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6813","nonce":173493555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6820","nonce":173493555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6816","nonce":173493555}]},"public_addr":"172.21.15.63:6810/173493555","cluster_addr":"172.21.15.63:6813/173493555","heartbeat_back_addr":"172.21.15.63:6820/173493555","heartbeat_front_addr":"172.21.15.63:6816/173493555","state":["exists","up"]},{"osd":6,"uuid":"db15cd66-2173-4e9a-9197-270c8ede9194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6825","nonce":3278865177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6827","nonce":3278865177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6831","nonce":3278865177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6829","nonce":3278865177}]},"public_addr":"172.21.15.63:6825/3278865177","cluster_addr":"172.21.15.63:6827/3278865177","heartbeat_back_addr":"172.21.15.63:6831/3278865177","heartbeat_front_addr":"172.21.15.63:6829/3278865177","state":["exists","up"]},{"osd":7,"uuid":"8b13f614-d1da-4b89-b91f-e530ce24febf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6801","nonce":2256787352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6803","nonce":2256787352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6807","nonce":2256787352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6805","nonce":2256787352}]},"public_addr":"172.21.15.63:6801/2256787352","cluster_addr":"172.21.15.63:6803/2256787352","heartbeat_back_addr":"172.21.15.63:6807/2256787352","heartbeat_front_addr":"172.21.15.63:6805/2256787352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.864765+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.821813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-01T12:52:12.085 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-01T12:52:12.085 INFO:tasks.ceph:Creating RBD pool 2024-05-01T12:52:12.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-01T12:52:12.895 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2024-05-01T12:52:12.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-01T12:52:13.900 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-01T12:52:13.913 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-01T12:52:13.914 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-01T12:52:13.914 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-01T12:52:13.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-01T12:52:13.915 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-01T12:52:14.272 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:14.282 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3466725479},{"type":"v1","addr":"172.21.15.50:6833","nonce":3466725479}]},"active_addr":"172.21.15.50:6833/3466725479","active_change":"2024-05-01T12:52:05.867400+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":758117082}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2473424081}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3710611519}]}]} 2024-05-01T12:52:14.288 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-01T12:52:14.288 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-01T12:52:14.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-01T12:52:14.587 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:14.588 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"fsid":"fcc845ba-0166-4b8a-a069-0a6a9d719b5c","created":"2024-05-01T12:52:01.606905+0000","modified":"2024-05-01T12:52:13.895841+0000","last_up_change":"2024-05-01T12:52:11.893267+0000","last_in_change":"2024-05-01T12:52:03.737913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-01T12:52:09.893222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-01T12:52:12.383710+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfbb253-e04e-42e1-af0a-c24d03aee610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6801","nonce":2305361892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6803","nonce":2305361892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6807","nonce":2305361892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6805","nonce":2305361892}]},"public_addr":"172.21.15.50:6801/2305361892","cluster_addr":"172.21.15.50:6803/2305361892","heartbeat_back_addr":"172.21.15.50:6807/2305361892","heartbeat_front_addr":"172.21.15.50:6805/2305361892","state":["exists","up"]},{"osd":1,"uuid":"85ce1d73-dcc0-43ca-ba94-e681b20d7053","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6825","nonce":247574176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6827","nonce":247574176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6831","nonce":247574176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6829","nonce":247574176}]},"public_addr":"172.21.15.50:6825/247574176","cluster_addr":"172.21.15.50:6827/247574176","heartbeat_back_addr":"172.21.15.50:6831/247574176","heartbeat_front_addr":"172.21.15.50:6829/247574176","state":["exists","up"]},{"osd":2,"uuid":"d7dac084-480e-4f7b-a6bf-194e27188ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6809","nonce":1822591763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6811","nonce":1822591763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6815","nonce":1822591763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6813","nonce":1822591763}]},"public_addr":"172.21.15.50:6809/1822591763","cluster_addr":"172.21.15.50:6811/1822591763","heartbeat_back_addr":"172.21.15.50:6815/1822591763","heartbeat_front_addr":"172.21.15.50:6813/1822591763","state":["exists","up"]},{"osd":3,"uuid":"96dd3a06-bcc8-4d86-8720-91a3d98b3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6817","nonce":2001127933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6819","nonce":2001127933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6823","nonce":2001127933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6821","nonce":2001127933}]},"public_addr":"172.21.15.50:6817/2001127933","cluster_addr":"172.21.15.50:6819/2001127933","heartbeat_back_addr":"172.21.15.50:6823/2001127933","heartbeat_front_addr":"172.21.15.50:6821/2001127933","state":["exists","up"]},{"osd":4,"uuid":"ec3a7c20-0e93-4431-96d7-24f75bc43093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6811","nonce":2421506900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6817","nonce":2421506900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6823","nonce":2421506900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6821","nonce":2421506900}]},"public_addr":"172.21.15.63:6811/2421506900","cluster_addr":"172.21.15.63:6817/2421506900","heartbeat_back_addr":"172.21.15.63:6823/2421506900","heartbeat_front_addr":"172.21.15.63:6821/2421506900","state":["exists","up"]},{"osd":5,"uuid":"a5420e0a-8bc2-4217-946c-1b4a3baa3e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6810","nonce":173493555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6813","nonce":173493555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6820","nonce":173493555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6816","nonce":173493555}]},"public_addr":"172.21.15.63:6810/173493555","cluster_addr":"172.21.15.63:6813/173493555","heartbeat_back_addr":"172.21.15.63:6820/173493555","heartbeat_front_addr":"172.21.15.63:6816/173493555","state":["exists","up"]},{"osd":6,"uuid":"db15cd66-2173-4e9a-9197-270c8ede9194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6825","nonce":3278865177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6827","nonce":3278865177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6831","nonce":3278865177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6829","nonce":3278865177}]},"public_addr":"172.21.15.63:6825/3278865177","cluster_addr":"172.21.15.63:6827/3278865177","heartbeat_back_addr":"172.21.15.63:6831/3278865177","heartbeat_front_addr":"172.21.15.63:6829/3278865177","state":["exists","up"]},{"osd":7,"uuid":"8b13f614-d1da-4b89-b91f-e530ce24febf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6801","nonce":2256787352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6803","nonce":2256787352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6807","nonce":2256787352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6805","nonce":2256787352}]},"public_addr":"172.21.15.63:6801/2256787352","cluster_addr":"172.21.15.63:6803/2256787352","heartbeat_back_addr":"172.21.15.63:6807/2256787352","heartbeat_front_addr":"172.21.15.63:6805/2256787352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.864765+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.849275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.862383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.835404+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.927275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.821813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.866460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.899200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-01T12:52:14.600 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-01T12:52:14.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-01T12:52:14.905 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:14.905 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":18,"fsid":"fcc845ba-0166-4b8a-a069-0a6a9d719b5c","created":"2024-05-01T12:52:01.606905+0000","modified":"2024-05-01T12:52:14.897006+0000","last_up_change":"2024-05-01T12:52:11.893267+0000","last_in_change":"2024-05-01T12:52:03.737913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-01T12:52:09.893222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-01T12:52:12.383710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfbb253-e04e-42e1-af0a-c24d03aee610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6801","nonce":2305361892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6803","nonce":2305361892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6807","nonce":2305361892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2305361892},{"type":"v1","addr":"172.21.15.50:6805","nonce":2305361892}]},"public_addr":"172.21.15.50:6801/2305361892","cluster_addr":"172.21.15.50:6803/2305361892","heartbeat_back_addr":"172.21.15.50:6807/2305361892","heartbeat_front_addr":"172.21.15.50:6805/2305361892","state":["exists","up"]},{"osd":1,"uuid":"85ce1d73-dcc0-43ca-ba94-e681b20d7053","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6825","nonce":247574176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6827","nonce":247574176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6831","nonce":247574176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":247574176},{"type":"v1","addr":"172.21.15.50:6829","nonce":247574176}]},"public_addr":"172.21.15.50:6825/247574176","cluster_addr":"172.21.15.50:6827/247574176","heartbeat_back_addr":"172.21.15.50:6831/247574176","heartbeat_front_addr":"172.21.15.50:6829/247574176","state":["exists","up"]},{"osd":2,"uuid":"d7dac084-480e-4f7b-a6bf-194e27188ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6809","nonce":1822591763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6811","nonce":1822591763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6815","nonce":1822591763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1822591763},{"type":"v1","addr":"172.21.15.50:6813","nonce":1822591763}]},"public_addr":"172.21.15.50:6809/1822591763","cluster_addr":"172.21.15.50:6811/1822591763","heartbeat_back_addr":"172.21.15.50:6815/1822591763","heartbeat_front_addr":"172.21.15.50:6813/1822591763","state":["exists","up"]},{"osd":3,"uuid":"96dd3a06-bcc8-4d86-8720-91a3d98b3558","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6817","nonce":2001127933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6819","nonce":2001127933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6823","nonce":2001127933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2001127933},{"type":"v1","addr":"172.21.15.50:6821","nonce":2001127933}]},"public_addr":"172.21.15.50:6817/2001127933","cluster_addr":"172.21.15.50:6819/2001127933","heartbeat_back_addr":"172.21.15.50:6823/2001127933","heartbeat_front_addr":"172.21.15.50:6821/2001127933","state":["exists","up"]},{"osd":4,"uuid":"ec3a7c20-0e93-4431-96d7-24f75bc43093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6811","nonce":2421506900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6817","nonce":2421506900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6823","nonce":2421506900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":2421506900},{"type":"v1","addr":"172.21.15.63:6821","nonce":2421506900}]},"public_addr":"172.21.15.63:6811/2421506900","cluster_addr":"172.21.15.63:6817/2421506900","heartbeat_back_addr":"172.21.15.63:6823/2421506900","heartbeat_front_addr":"172.21.15.63:6821/2421506900","state":["exists","up"]},{"osd":5,"uuid":"a5420e0a-8bc2-4217-946c-1b4a3baa3e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6810","nonce":173493555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6813","nonce":173493555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6820","nonce":173493555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":173493555},{"type":"v1","addr":"172.21.15.63:6816","nonce":173493555}]},"public_addr":"172.21.15.63:6810/173493555","cluster_addr":"172.21.15.63:6813/173493555","heartbeat_back_addr":"172.21.15.63:6820/173493555","heartbeat_front_addr":"172.21.15.63:6816/173493555","state":["exists","up"]},{"osd":6,"uuid":"db15cd66-2173-4e9a-9197-270c8ede9194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6825","nonce":3278865177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6827","nonce":3278865177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6831","nonce":3278865177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":3278865177},{"type":"v1","addr":"172.21.15.63:6829","nonce":3278865177}]},"public_addr":"172.21.15.63:6825/3278865177","cluster_addr":"172.21.15.63:6827/3278865177","heartbeat_back_addr":"172.21.15.63:6831/3278865177","heartbeat_front_addr":"172.21.15.63:6829/3278865177","state":["exists","up"]},{"osd":7,"uuid":"8b13f614-d1da-4b89-b91f-e530ce24febf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6801","nonce":2256787352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6803","nonce":2256787352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6807","nonce":2256787352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2256787352},{"type":"v1","addr":"172.21.15.63:6805","nonce":2256787352}]},"public_addr":"172.21.15.63:6801/2256787352","cluster_addr":"172.21.15.63:6803/2256787352","heartbeat_back_addr":"172.21.15.63:6807/2256787352","heartbeat_front_addr":"172.21.15.63:6805/2256787352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.864765+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.849275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.862383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.835404+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.927275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.821813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.866460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-01T12:52:08.899200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-01T12:52:14.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-01T12:52:14.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-01T12:52:14.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-01T12:52:14.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-01T12:52:14.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-01T12:52:14.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-01T12:52:14.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-01T12:52:14.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-01T12:52:15.167 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-01T12:52:15.172 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:15.172 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:15.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-01T12:52:15.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-01T12:52:15.175 INFO:teuthology.orchestra.run.smithi050.stdout:55834574851 2024-05-01T12:52:15.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-01T12:52:15.177 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:15.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-01T12:52:15.178 INFO:teuthology.orchestra.run.smithi050.stdout:55834574852 2024-05-01T12:52:15.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-01T12:52:15.180 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-01T12:52:15.182 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:15.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-01T12:52:15.763 INFO:teuthology.orchestra.run.smithi050.stdout:64424509442 2024-05-01T12:52:15.772 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.777 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.783 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-05-01T12:52:15.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.6 2024-05-01T12:52:15.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.7 2024-05-01T12:52:15.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.3 2024-05-01T12:52:15.808 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 1 for osd.0 2024-05-01T12:52:15.816 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.819 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:15.823 INFO:teuthology.orchestra.run.smithi050.stdout:55834574850 2024-05-01T12:52:15.825 INFO:teuthology.orchestra.run.smithi050.stdout:64424509442 2024-05-01T12:52:15.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.4 2024-05-01T12:52:15.837 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.1 2024-05-01T12:52:15.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.5 2024-05-01T12:52:15.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.2 2024-05-01T12:52:16.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-01T12:52:16.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-01T12:52:16.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-01T12:52:16.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-01T12:52:16.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-01T12:52:16.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-01T12:52:16.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-01T12:52:16.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-01T12:52:17.366 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:17.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.6 2024-05-01T12:52:17.386 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.417 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:17.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.7 2024-05-01T12:52:17.431 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.436 INFO:teuthology.orchestra.run.smithi050.stdout:55834574851 2024-05-01T12:52:17.442 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:17.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2024-05-01T12:52:17.455 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.3 2024-05-01T12:52:17.461 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.475 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:17.484 INFO:teuthology.orchestra.run.smithi050.stdout:64424509443 2024-05-01T12:52:17.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.4 2024-05-01T12:52:17.489 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.2 2024-05-01T12:52:17.497 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.512 INFO:teuthology.orchestra.run.smithi050.stdout:64424509444 2024-05-01T12:52:17.519 INFO:teuthology.orchestra.run.smithi050.stdout:55834574852 2024-05-01T12:52:17.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.1 2024-05-01T12:52:17.526 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-05-01T12:52:17.532 DEBUG:teuthology.parallel:result is None 2024-05-01T12:52:17.533 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-01T12:52:17.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-01T12:52:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-05-01T12:52:17.880 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-05-01T12:52:15.874217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243648,"kb_used_data":2408,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754731072,"statfs":{"total":773094113280,"available":772844617728,"internally_reserved":0,"allocated":2465792,"data_stored":1575492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900533+0000","last_change":"2024-05-01T12:52:14.716206+0000","last_active":"2024-05-01T12:52:14.900533+0000","last_peered":"2024-05-01T12:52:14.900533+0000","last_clean":"2024-05-01T12:52:14.900533+0000","last_became_active":"2024-05-01T12:52:14.715875+0000","last_became_peered":"2024-05-01T12:52:14.715875+0000","last_unstale":"2024-05-01T12:52:14.900533+0000","last_undegraded":"2024-05-01T12:52:14.900533+0000","last_fullsized":"2024-05-01T12:52:14.900533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T23:36:05.145509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.915143+0000","last_change":"2024-05-01T12:52:14.716466+0000","last_active":"2024-05-01T12:52:14.915143+0000","last_peered":"2024-05-01T12:52:14.915143+0000","last_clean":"2024-05-01T12:52:14.915143+0000","last_became_active":"2024-05-01T12:52:14.716148+0000","last_became_peered":"2024-05-01T12:52:14.716148+0000","last_unstale":"2024-05-01T12:52:14.915143+0000","last_undegraded":"2024-05-01T12:52:14.915143+0000","last_fullsized":"2024-05-01T12:52:14.915143+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:21.403749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900403+0000","last_change":"2024-05-01T12:52:13.903521+0000","last_active":"2024-05-01T12:52:14.900403+0000","last_peered":"2024-05-01T12:52:14.900403+0000","last_clean":"2024-05-01T12:52:14.900403+0000","last_became_active":"2024-05-01T12:52:13.903252+0000","last_became_peered":"2024-05-01T12:52:13.903252+0000","last_unstale":"2024-05-01T12:52:14.900403+0000","last_undegraded":"2024-05-01T12:52:14.900403+0000","last_fullsized":"2024-05-01T12:52:14.900403+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T16:41:44.278413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.915463+0000","last_change":"2024-05-01T12:52:14.292301+0000","last_active":"2024-05-01T12:52:14.915463+0000","last_peered":"2024-05-01T12:52:14.915463+0000","last_clean":"2024-05-01T12:52:14.915463+0000","last_became_active":"2024-05-01T12:52:14.291908+0000","last_became_peered":"2024-05-01T12:52:14.291908+0000","last_unstale":"2024-05-01T12:52:14.915463+0000","last_undegraded":"2024-05-01T12:52:14.915463+0000","last_fullsized":"2024-05-01T12:52:14.915463+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-03T00:19:58.170014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900392+0000","last_change":"2024-05-01T12:52:14.292370+0000","last_active":"2024-05-01T12:52:14.900392+0000","last_peered":"2024-05-01T12:52:14.900392+0000","last_clean":"2024-05-01T12:52:14.900392+0000","last_became_active":"2024-05-01T12:52:14.292046+0000","last_became_peered":"2024-05-01T12:52:14.292046+0000","last_unstale":"2024-05-01T12:52:14.900392+0000","last_undegraded":"2024-05-01T12:52:14.900392+0000","last_fullsized":"2024-05-01T12:52:14.900392+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T18:50:57.716627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900085+0000","last_change":"2024-05-01T12:52:13.904140+0000","last_active":"2024-05-01T12:52:14.900085+0000","last_peered":"2024-05-01T12:52:14.900085+0000","last_clean":"2024-05-01T12:52:14.900085+0000","last_became_active":"2024-05-01T12:52:13.903623+0000","last_became_peered":"2024-05-01T12:52:13.903623+0000","last_unstale":"2024-05-01T12:52:14.900085+0000","last_undegraded":"2024-05-01T12:52:14.900085+0000","last_fullsized":"2024-05-01T12:52:14.900085+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:21.403749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900372+0000","last_change":"2024-05-01T12:52:13.903849+0000","last_active":"2024-05-01T12:52:14.900372+0000","last_peered":"2024-05-01T12:52:14.900372+0000","last_clean":"2024-05-01T12:52:14.900372+0000","last_became_active":"2024-05-01T12:52:13.903552+0000","last_became_peered":"2024-05-01T12:52:13.903552+0000","last_unstale":"2024-05-01T12:52:14.900372+0000","last_undegraded":"2024-05-01T12:52:14.900372+0000","last_fullsized":"2024-05-01T12:52:14.900372+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T20:27:45.729142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900385+0000","last_change":"2024-05-01T12:52:14.291159+0000","last_active":"2024-05-01T12:52:14.900385+0000","last_peered":"2024-05-01T12:52:14.900385+0000","last_clean":"2024-05-01T12:52:14.900385+0000","last_became_active":"2024-05-01T12:52:14.290901+0000","last_became_peered":"2024-05-01T12:52:14.290901+0000","last_unstale":"2024-05-01T12:52:14.900385+0000","last_undegraded":"2024-05-01T12:52:14.900385+0000","last_fullsized":"2024-05-01T12:52:14.900385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T22:25:45.271253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900128+0000","last_change":"2024-05-01T12:52:12.904883+0000","last_active":"2024-05-01T12:52:14.900128+0000","last_peered":"2024-05-01T12:52:14.900128+0000","last_clean":"2024-05-01T12:52:14.900128+0000","last_became_active":"2024-05-01T12:52:12.903696+0000","last_became_peered":"2024-05-01T12:52:12.903696+0000","last_unstale":"2024-05-01T12:52:14.900128+0000","last_undegraded":"2024-05-01T12:52:14.900128+0000","last_fullsized":"2024-05-01T12:52:14.900128+0000","mapping_epoch":15,"log_start":"16'100","ondisk_log_start":"16'100","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:10.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:10.882556+0000","last_clean_scrub_stamp":"2024-05-01T12:52:10.882556+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:19.391741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":172032,"data_stored":55149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":753664,"data_stored":639769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":753664,"data_stored":639769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35484,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336356,"statfs":{"total":96636764160,"available":96600428544,"internally_reserved":0,"allocated":147456,"data_stored":43201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-01T12:52:17.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-01T12:52:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-05-01T12:52:18.189 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-05-01T12:52:17.874510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243648,"kb_used_data":2408,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754731072,"statfs":{"total":773094113280,"available":772844617728,"internally_reserved":0,"allocated":2465792,"data_stored":1575492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000292"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900533+0000","last_change":"2024-05-01T12:52:14.716206+0000","last_active":"2024-05-01T12:52:14.900533+0000","last_peered":"2024-05-01T12:52:14.900533+0000","last_clean":"2024-05-01T12:52:14.900533+0000","last_became_active":"2024-05-01T12:52:14.715875+0000","last_became_peered":"2024-05-01T12:52:14.715875+0000","last_unstale":"2024-05-01T12:52:14.900533+0000","last_undegraded":"2024-05-01T12:52:14.900533+0000","last_fullsized":"2024-05-01T12:52:14.900533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T23:36:05.145509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.915143+0000","last_change":"2024-05-01T12:52:14.716466+0000","last_active":"2024-05-01T12:52:14.915143+0000","last_peered":"2024-05-01T12:52:14.915143+0000","last_clean":"2024-05-01T12:52:14.915143+0000","last_became_active":"2024-05-01T12:52:14.716148+0000","last_became_peered":"2024-05-01T12:52:14.716148+0000","last_unstale":"2024-05-01T12:52:14.915143+0000","last_undegraded":"2024-05-01T12:52:14.915143+0000","last_fullsized":"2024-05-01T12:52:14.915143+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:21.403749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900403+0000","last_change":"2024-05-01T12:52:13.903521+0000","last_active":"2024-05-01T12:52:14.900403+0000","last_peered":"2024-05-01T12:52:14.900403+0000","last_clean":"2024-05-01T12:52:14.900403+0000","last_became_active":"2024-05-01T12:52:13.903252+0000","last_became_peered":"2024-05-01T12:52:13.903252+0000","last_unstale":"2024-05-01T12:52:14.900403+0000","last_undegraded":"2024-05-01T12:52:14.900403+0000","last_fullsized":"2024-05-01T12:52:14.900403+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T16:41:44.278413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.915463+0000","last_change":"2024-05-01T12:52:14.292301+0000","last_active":"2024-05-01T12:52:14.915463+0000","last_peered":"2024-05-01T12:52:14.915463+0000","last_clean":"2024-05-01T12:52:14.915463+0000","last_became_active":"2024-05-01T12:52:14.291908+0000","last_became_peered":"2024-05-01T12:52:14.291908+0000","last_unstale":"2024-05-01T12:52:14.915463+0000","last_undegraded":"2024-05-01T12:52:14.915463+0000","last_fullsized":"2024-05-01T12:52:14.915463+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-03T00:19:58.170014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900392+0000","last_change":"2024-05-01T12:52:14.292370+0000","last_active":"2024-05-01T12:52:14.900392+0000","last_peered":"2024-05-01T12:52:14.900392+0000","last_clean":"2024-05-01T12:52:14.900392+0000","last_became_active":"2024-05-01T12:52:14.292046+0000","last_became_peered":"2024-05-01T12:52:14.292046+0000","last_unstale":"2024-05-01T12:52:14.900392+0000","last_undegraded":"2024-05-01T12:52:14.900392+0000","last_fullsized":"2024-05-01T12:52:14.900392+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T18:50:57.716627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900085+0000","last_change":"2024-05-01T12:52:13.904140+0000","last_active":"2024-05-01T12:52:14.900085+0000","last_peered":"2024-05-01T12:52:14.900085+0000","last_clean":"2024-05-01T12:52:14.900085+0000","last_became_active":"2024-05-01T12:52:13.903623+0000","last_became_peered":"2024-05-01T12:52:13.903623+0000","last_unstale":"2024-05-01T12:52:14.900085+0000","last_undegraded":"2024-05-01T12:52:14.900085+0000","last_fullsized":"2024-05-01T12:52:14.900085+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:21.403749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900372+0000","last_change":"2024-05-01T12:52:13.903849+0000","last_active":"2024-05-01T12:52:14.900372+0000","last_peered":"2024-05-01T12:52:14.900372+0000","last_clean":"2024-05-01T12:52:14.900372+0000","last_became_active":"2024-05-01T12:52:13.903552+0000","last_became_peered":"2024-05-01T12:52:13.903552+0000","last_unstale":"2024-05-01T12:52:14.900372+0000","last_undegraded":"2024-05-01T12:52:14.900372+0000","last_fullsized":"2024-05-01T12:52:14.900372+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T20:27:45.729142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900385+0000","last_change":"2024-05-01T12:52:14.291159+0000","last_active":"2024-05-01T12:52:14.900385+0000","last_peered":"2024-05-01T12:52:14.900385+0000","last_clean":"2024-05-01T12:52:14.900385+0000","last_became_active":"2024-05-01T12:52:14.290901+0000","last_became_peered":"2024-05-01T12:52:14.290901+0000","last_unstale":"2024-05-01T12:52:14.900385+0000","last_undegraded":"2024-05-01T12:52:14.900385+0000","last_fullsized":"2024-05-01T12:52:14.900385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:12.894564+0000","last_clean_scrub_stamp":"2024-05-01T12:52:12.894564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T22:25:45.271253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-05-01T12:52:14.900128+0000","last_change":"2024-05-01T12:52:12.904883+0000","last_active":"2024-05-01T12:52:14.900128+0000","last_peered":"2024-05-01T12:52:14.900128+0000","last_clean":"2024-05-01T12:52:14.900128+0000","last_became_active":"2024-05-01T12:52:12.903696+0000","last_became_peered":"2024-05-01T12:52:12.903696+0000","last_unstale":"2024-05-01T12:52:14.900128+0000","last_undegraded":"2024-05-01T12:52:14.900128+0000","last_fullsized":"2024-05-01T12:52:14.900128+0000","mapping_epoch":15,"log_start":"16'100","ondisk_log_start":"16'100","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-01T12:52:10.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-01T12:52:10.882556+0000","last_clean_scrub_stamp":"2024-05-01T12:52:10.882556+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-02T15:54:19.391741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":172032,"data_stored":55149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":159744,"data_stored":49401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":753664,"data_stored":639769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":753664,"data_stored":639769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35484,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336356,"statfs":{"total":96636764160,"available":96600428544,"internally_reserved":0,"allocated":147456,"data_stored":43201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-01T12:52:18.190 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-01T12:52:18.190 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-01T12:52:18.190 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-01T12:52:18.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-01T12:52:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:52:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-01T12:52:18.537 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-01T12:52:18.538 INFO:teuthology.run_tasks:Running task openssl_keys... 2024-05-01T12:52:18.549 INFO:teuthology.run_tasks:Running task rgw... 2024-05-01T12:52:18.572 DEBUG:tasks.rgw:config is {'client.0': None, 'client.1': None} 2024-05-01T12:52:18.572 DEBUG:tasks.rgw:client list is dict_keys(['client.0', 'client.1']) 2024-05-01T12:52:18.572 INFO:tasks.rgw:Creating data pools 2024-05-01T12:52:18.572 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-05-01T12:52:18.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create MyZone.rgw.buckets.data 64 64 --cluster ceph 2024-05-01T12:52:19.138 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'MyZone.rgw.buckets.data' created 2024-05-01T12:52:19.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable MyZone.rgw.buckets.data rgw --cluster ceph 2024-05-01T12:52:20.143 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.data' 2024-05-01T12:52:20.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create MyZone.rgw.buckets.index 64 64 --cluster ceph 2024-05-01T12:52:21.148 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'MyZone.rgw.buckets.index' created 2024-05-01T12:52:21.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable MyZone.rgw.buckets.index rgw --cluster ceph 2024-05-01T12:52:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.index' 2024-05-01T12:52:22.188 DEBUG:tasks.rgw:Obtaining remote for client client.1 2024-05-01T12:52:22.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create MyZone.rgw.buckets.data 64 64 --cluster ceph 2024-05-01T12:52:22.494 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'MyZone.rgw.buckets.data' already exists 2024-05-01T12:52:22.506 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable MyZone.rgw.buckets.data rgw --cluster ceph 2024-05-01T12:52:23.157 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.data' 2024-05-01T12:52:23.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create MyZone.rgw.buckets.index 64 64 --cluster ceph 2024-05-01T12:52:23.472 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'MyZone.rgw.buckets.index' already exists 2024-05-01T12:52:23.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable MyZone.rgw.buckets.index rgw --cluster ceph 2024-05-01T12:52:24.167 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'MyZone.rgw.buckets.index' 2024-05-01T12:52:24.180 DEBUG:tasks.rgw:Pools created 2024-05-01T12:52:24.180 INFO:tasks.rgw:Creating realm MyRealm 2024-05-01T12:52:24.180 INFO:tasks.util.rgw:rgwadmin: client.0 : ['realm', 'create', '--rgw-realm', 'MyRealm', '--default'] 2024-05-01T12:52:24.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'realm', 'create', '--rgw-realm', 'MyRealm', '--default'] 2024-05-01T12:52:24.181 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph realm create --rgw-realm MyRealm --default 2024-05-01T12:52:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:24.288+0000 7f12f3729800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:24.288+0000 7f12f3729800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:24.292+0000 7f1165ffb640 20 reqs_thread_entry: start 2024-05-01T12:52:26.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.206+0000 7f12f3729800 20 creating initial latest_epoch=1 for period=5c1e957f-be5d-4a49-9a25-a52c4382e70f 2024-05-01T12:52:26.240 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f","name":"MyRealm","current_period":"5c1e957f-be5d-4a49-9a25-a52c4382e70f","epoch":1} 2024-05-01T12:52:26.240 DEBUG:tasks.util.rgw: json result: {'id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f', 'name': 'MyRealm', 'current_period': '5c1e957f-be5d-4a49-9a25-a52c4382e70f', 'epoch': 1} 2024-05-01T12:52:26.240 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--master', '--default'] 2024-05-01T12:52:26.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--master', '--default'] 2024-05-01T12:52:26.240 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup create --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --master --default 2024-05-01T12:52:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:26.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.316+0000 7f29c783f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:26.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.316+0000 7f29c783f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.320+0000 7f27fbfff640 20 reqs_thread_entry: start 2024-05-01T12:52:26.344 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"03460ef8-bf75-4e87-8999-38b335ab7ab6","name":"MyZoneGroup","api_name":"MyZoneGroup","is_master":true,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":[]}],"default_placement":"default-placement","realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2024-05-01T12:52:26.345 DEBUG:tasks.util.rgw: json result: {'id': '03460ef8-bf75-4e87-8999-38b335ab7ab6', 'name': 'MyZoneGroup', 'api_name': 'MyZoneGroup', 'is_master': True, 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': '', 'zones': [], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': []}], 'default_placement': 'default-placement', 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f', 'sync_policy': {'groups': []}, 'enabled_features': ['notification_v2', 'resharding']} 2024-05-01T12:52:26.345 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone', '--master', '--default'] 2024-05-01T12:52:26.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'create', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone', '--master', '--default'] 2024-05-01T12:52:26.345 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone create --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --rgw-zone MyZone --master --default 2024-05-01T12:52:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:26.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.417+0000 7ff8b9413800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:26.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.417+0000 7ff8b9413800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:26.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.420+0000 7ff6faffd640 20 reqs_thread_entry: start 2024-05-01T12:52:26.423 INFO:teuthology.orchestra.run.smithi050.stderr:NOTICE: set zone's realm_id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.453 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f"} 2024-05-01T12:52:26.453 DEBUG:tasks.util.rgw: json result: {'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f'} 2024-05-01T12:52:26.453 INFO:tasks.util.rgw:rgwadmin: client.0 : ['period', 'update', '--commit', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone'] 2024-05-01T12:52:26.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'period', 'update', '--commit', '--rgw-realm', 'MyRealm', '--rgw-zonegroup', 'MyZoneGroup', '--rgw-zone', 'MyZone'] 2024-05-01T12:52:26.454 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph period update --commit --rgw-realm MyRealm --rgw-zonegroup MyZoneGroup --rgw-zone MyZone 2024-05-01T12:52:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:26.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.532+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.534+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:52:26.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.534+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.535+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:52:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.535+0000 7fd71a92f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.535+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.535+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:52:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.535+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.536+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:52:26.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.536+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.536+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=359 2024-05-01T12:52:26.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.536+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.537+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:52:26.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.537+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.538+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=1005 2024-05-01T12:52:26.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.538+0000 7fd71a92f800 0 period (5c1e957f-be5d-4a49-9a25-a52c4382e70f does not have zone b3b7ffad-8c5b-4f40-9e07-35170fac4823 configured 2024-05-01T12:52:26.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.538+0000 7fd71a92f800 20 searching for the correct realm 2024-05-01T12:52:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got default.zone.ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got periods.5c1e957f-be5d-4a49-9a25-a52c4382e70f.1 2024-05-01T12:52:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got zone_info.b3b7ffad-8c5b-4f40-9e07-35170fac4823 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got default.realm 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got zone_names.MyZone 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got period_config.ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got realms.ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got zonegroup_info.03460ef8-bf75-4e87-8999-38b335ab7ab6 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got realms.ecdbd3a5-899c-4649-8e14-01e69eca1c7f.control 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got default.zonegroup.ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got periods.5c1e957f-be5d-4a49-9a25-a52c4382e70f.latest_epoch 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got realms_names.MyRealm 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 RGWRados::pool_iterate: got zonegroups_names.MyZoneGroup 2024-05-01T12:52:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.549+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.550+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=359 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=517 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 zone MyZone found 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 10 cannot find current period zonegroup using local zonegroup configuration 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 zonegroup MyZoneGroup 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados_obj.operate() r=0 bl.length=187 2024-05-01T12:52:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:26.551+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:28.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:28.196+0000 7fd71a92f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:52:28.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:28.196+0000 7fd71a92f800 20 rados->read ofs=0 len=0 2024-05-01T12:52:28.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:28.199+0000 7fd71a92f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:52:28.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:28.199+0000 7fd71a92f800 20 started sync module instance, tier type = 2024-05-01T12:52:28.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:28.199+0000 7fd71a92f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=0 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=1 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=2 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=3 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=4 2024-05-01T12:52:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=5 2024-05-01T12:52:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.222+0000 7fd71a92f800 20 add_watcher() i=6 2024-05-01T12:52:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.225+0000 7fd71a92f800 20 add_watcher() i=7 2024-05-01T12:52:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.225+0000 7fd71a92f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:52:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-05-01T12:52:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-05-01T12:52:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-05-01T12:52:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-05-01T12:52:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.226+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-05-01T12:52:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.227+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2024-05-01T12:52:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2024-05-01T12:52:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.228+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2024-05-01T12:52:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.229+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.230+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2024-05-01T12:52:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2024-05-01T12:52:30.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.231+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2024-05-01T12:52:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.232+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.233+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2024-05-01T12:52:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.234+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2024-05-01T12:52:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.235+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2024-05-01T12:52:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2024-05-01T12:52:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.236+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.237+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.238+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2024-05-01T12:52:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.239+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.240+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.241+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.242+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.243+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2024-05-01T12:52:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.244+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2024-05-01T12:52:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.245+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2024-05-01T12:52:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2024-05-01T12:52:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.246+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2024-05-01T12:52:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.247+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.248+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2024-05-01T12:52:30.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2024-05-01T12:52:30.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.249+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.250+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2024-05-01T12:52:30.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2024-05-01T12:52:30.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.251+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.252+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2024-05-01T12:52:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2024-05-01T12:52:30.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.253+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2024-05-01T12:52:30.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.254+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2024-05-01T12:52:30.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.255+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2024-05-01T12:52:30.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.256+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2024-05-01T12:52:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.257+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2024-05-01T12:52:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.258+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2024-05-01T12:52:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.259+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2024-05-01T12:52:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.260+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1211 entering 2024-05-01T12:52:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.261+0000 7fd71a92f800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1169 entering 2024-05-01T12:52:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.264+0000 7fd5ccfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:52:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.264+0000 7fd71a92f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:30.264+0000 7fd71a92f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.190+0000 7fd71a92f800 10 rgw_init_ioctx warning: failed to set recovery_priority on MyZone.rgw.meta 2024-05-01T12:52:33.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.190+0000 7fd71a92f800 5 note: GC not initialized 2024-05-01T12:52:33.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.191+0000 7fd5377fe640 20 reqs_thread_entry: start 2024-05-01T12:52:33.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.191+0000 7fd71a92f800 20 init_complete bucket index max shards: 11 2024-05-01T12:52:33.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.191+0000 7fd71a92f800 20 Filter name: none 2024-05-01T12:52:33.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.192+0000 7fd5357fa640 20 reqs_thread_entry: start 2024-05-01T12:52:33.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.194+0000 7fd71a92f800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=5c1e957f-be5d-4a49-9a25-a52c4382e70f 2024-05-01T12:52:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.229+0000 7fd71a92f800 20 creating initial latest_epoch=1 for period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T12:52:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.231+0000 7fd71a92f800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 5c1e957f-be5d-4a49-9a25-a52c4382e70f 2024-05-01T12:52:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.233+0000 7fd71a92f800 20 creating initial latest_epoch=1 for period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T12:52:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.244+0000 7fd71a92f800 4 Promoted to master zone and committed new period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T12:52:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.245+0000 7fd5ccfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:52:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.248+0000 7fd71a92f800 20 remove_watcher() i=0 2024-05-01T12:52:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.248+0000 7fd71a92f800 2 removed watcher, disabling cache 2024-05-01T12:52:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.249+0000 7fd71a92f800 20 remove_watcher() i=1 2024-05-01T12:52:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.251+0000 7fd71a92f800 20 remove_watcher() i=2 2024-05-01T12:52:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.252+0000 7fd71a92f800 20 remove_watcher() i=3 2024-05-01T12:52:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.253+0000 7fd71a92f800 20 remove_watcher() i=4 2024-05-01T12:52:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.255+0000 7fd71a92f800 20 remove_watcher() i=5 2024-05-01T12:52:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.256+0000 7fd71a92f800 20 remove_watcher() i=6 2024-05-01T12:52:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.257+0000 7fd71a92f800 20 remove_watcher() i=7 2024-05-01T12:52:33.269 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"511d2184-89cf-4665-9417-968dd02e356b","epoch":1,"predecessor_uuid":"5c1e957f-be5d-4a49-9a25-a52c4382e70f","sync_status":[],"period_map":{"id":"511d2184-89cf-4665-9417-968dd02e356b","zonegroups":[{"id":"03460ef8-bf75-4e87-8999-38b335ab7ab6","name":"MyZoneGroup","api_name":"MyZoneGroup","is_master":true,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","zones":[{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["compress-encrypted","notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]}],"short_zone_ids":[{"key":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","val":2512848662}]},"master_zonegroup":"03460ef8-bf75-4e87-8999-38b335ab7ab6","master_zone":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"bucket_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false},"anonymous_ratelimit":{"max_read_ops":0,"max_write_ops":0,"max_read_bytes":0,"max_write_bytes":0,"enabled":false}},"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f","realm_epoch":2} 2024-05-01T12:52:33.269 DEBUG:tasks.util.rgw: json result: {'id': '511d2184-89cf-4665-9417-968dd02e356b', 'epoch': 1, 'predecessor_uuid': '5c1e957f-be5d-4a49-9a25-a52c4382e70f', 'sync_status': [], 'period_map': {'id': '511d2184-89cf-4665-9417-968dd02e356b', 'zonegroups': [{'id': '03460ef8-bf75-4e87-8999-38b335ab7ab6', 'name': 'MyZoneGroup', 'api_name': 'MyZoneGroup', 'is_master': True, 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'zones': [{'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'endpoints': [], 'log_meta': False, 'log_data': False, 'bucket_index_max_shards': 11, 'read_only': False, 'tier_type': '', 'sync_from_all': True, 'sync_from': [], 'redirect_zone': '', 'supported_features': ['compress-encrypted', 'notification_v2', 'resharding']}], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': ['STANDARD']}], 'default_placement': 'default-placement', 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f', 'sync_policy': {'groups': []}, 'enabled_features': ['notification_v2', 'resharding']}], 'short_zone_ids': [{'key': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'val': 2512848662}]}, 'master_zonegroup': '03460ef8-bf75-4e87-8999-38b335ab7ab6', 'master_zone': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'period_config': {'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'anonymous_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}}, 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f', 'realm_epoch': 2} 2024-05-01T12:52:33.269 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2024-05-01T12:52:33.270 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-05-01T12:52:33.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-05-01T12:52:33.270 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM 2024-05-01T12:52:33.320 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.320 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.345+0000 7f2941f23800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.345+0000 7f2941f23800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.349+0000 7f2773fff640 20 reqs_thread_entry: start 2024-05-01T12:52:33.364 INFO:teuthology.orchestra.run.smithi050.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2024-05-01T12:52:33.364 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2024-05-01T12:52:33.364 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-05-01T12:52:33.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-05-01T12:52:33.364 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2024-05-01T12:52:33.414 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.415 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.438+0000 7effcfa3f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.438+0000 7effcfa3f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.442+0000 7efe0b7fe640 20 reqs_thread_entry: start 2024-05-01T12:52:33.458 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f"} 2024-05-01T12:52:33.459 DEBUG:tasks.util.rgw: json result: {'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f'} 2024-05-01T12:52:33.459 INFO:tasks.rgw:Configuring storage class type = FROZEN 2024-05-01T12:52:33.459 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-05-01T12:52:33.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-05-01T12:52:33.459 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN 2024-05-01T12:52:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.530+0000 7fe11db35800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.530+0000 7fe11db35800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.534+0000 7fdf577fe640 20 reqs_thread_entry: start 2024-05-01T12:52:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2024-05-01T12:52:33.549 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-05-01T12:52:33.549 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-05-01T12:52:33.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-05-01T12:52:33.550 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2024-05-01T12:52:33.600 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.600 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.624+0000 7fd536a38800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.624+0000 7fd536a38800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:52:33.628+0000 7fd3f2ffd640 20 reqs_thread_entry: start 2024-05-01T12:52:33.645 INFO:teuthology.orchestra.run.smithi050.stdout:{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f"} 2024-05-01T12:52:33.645 DEBUG:tasks.util.rgw: json result: {'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f'} 2024-05-01T12:52:33.645 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2024-05-01T12:52:33.645 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-05-01T12:52:33.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-05-01T12:52:33.646 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM 2024-05-01T12:52:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.753+0000 7f6334536800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.753+0000 7f6334536800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.757+0000 7f619e7fc640 20 reqs_thread_entry: start 2024-05-01T12:52:33.792 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2024-05-01T12:52:33.792 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-05-01T12:52:33.792 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-05-01T12:52:33.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-05-01T12:52:33.792 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2024-05-01T12:52:33.844 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.844 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.869+0000 7f2ce7813800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.869+0000 7f2ce7813800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.872+0000 7f2ba0ff9640 20 reqs_thread_entry: start 2024-05-01T12:52:33.888 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f"} 2024-05-01T12:52:33.888 DEBUG:tasks.util.rgw: json result: {'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f'} 2024-05-01T12:52:33.889 INFO:tasks.rgw:Configuring storage class type = FROZEN 2024-05-01T12:52:33.889 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-05-01T12:52:33.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-05-01T12:52:33.889 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zonegroup placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN 2024-05-01T12:52:33.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:33.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:33.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.960+0000 7feaad213800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:33.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.960+0000 7feaad213800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:33.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:33.964+0000 7fe8eeffd640 20 reqs_thread_entry: start 2024-05-01T12:52:33.978 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2024-05-01T12:52:33.978 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-05-01T12:52:33.978 INFO:tasks.util.rgw:rgwadmin: client.1 : ['zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-05-01T12:52:33.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'MyZone', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-05-01T12:52:33.979 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster ceph zone placement add --rgw-zone MyZone --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2024-05-01T12:52:34.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:52:34.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:52:34.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:34.052+0000 7f5347f26800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:52:34.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:34.052+0000 7f5347f26800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:52:34.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-05-01T12:52:34.056+0000 7f5201ffb640 20 reqs_thread_entry: start 2024-05-01T12:52:34.072 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3b7ffad-8c5b-4f40-9e07-35170fac4823","name":"MyZone","domain_root":"MyZone.rgw.meta:root","control_pool":"MyZone.rgw.control","gc_pool":"MyZone.rgw.log:gc","lc_pool":"MyZone.rgw.log:lc","log_pool":"MyZone.rgw.log","intent_log_pool":"MyZone.rgw.log:intent","usage_log_pool":"MyZone.rgw.log:usage","roles_pool":"MyZone.rgw.meta:roles","reshard_pool":"MyZone.rgw.log:reshard","user_keys_pool":"MyZone.rgw.meta:users.keys","user_email_pool":"MyZone.rgw.meta:users.email","user_swift_pool":"MyZone.rgw.meta:users.swift","user_uid_pool":"MyZone.rgw.meta:users.uid","otp_pool":"MyZone.rgw.otp","notif_pool":"MyZone.rgw.log:notif","topics_pool":"MyZone.rgw.meta:topics","account_pool":"MyZone.rgw.meta:accounts","group_pool":"MyZone.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"MyZone.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"MyZone.rgw.buckets.data"}},"data_extra_pool":"MyZone.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"ecdbd3a5-899c-4649-8e14-01e69eca1c7f"} 2024-05-01T12:52:34.072 DEBUG:tasks.util.rgw: json result: {'id': 'b3b7ffad-8c5b-4f40-9e07-35170fac4823', 'name': 'MyZone', 'domain_root': 'MyZone.rgw.meta:root', 'control_pool': 'MyZone.rgw.control', 'gc_pool': 'MyZone.rgw.log:gc', 'lc_pool': 'MyZone.rgw.log:lc', 'log_pool': 'MyZone.rgw.log', 'intent_log_pool': 'MyZone.rgw.log:intent', 'usage_log_pool': 'MyZone.rgw.log:usage', 'roles_pool': 'MyZone.rgw.meta:roles', 'reshard_pool': 'MyZone.rgw.log:reshard', 'user_keys_pool': 'MyZone.rgw.meta:users.keys', 'user_email_pool': 'MyZone.rgw.meta:users.email', 'user_swift_pool': 'MyZone.rgw.meta:users.swift', 'user_uid_pool': 'MyZone.rgw.meta:users.uid', 'otp_pool': 'MyZone.rgw.otp', 'notif_pool': 'MyZone.rgw.log:notif', 'topics_pool': 'MyZone.rgw.meta:topics', 'account_pool': 'MyZone.rgw.meta:accounts', 'group_pool': 'MyZone.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'MyZone.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'MyZone.rgw.buckets.data'}}, 'data_extra_pool': 'MyZone.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': 'ecdbd3a5-899c-4649-8e14-01e69eca1c7f'} 2024-05-01T12:52:34.072 INFO:tasks.rgw:Starting rgw... 2024-05-01T12:52:34.072 INFO:tasks.rgw:rgw client.0 config is {} 2024-05-01T12:52:34.072 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-01T12:52:34.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo echo -n http://smithi050.front.sepia.ceph.com:80 | sudo tee /home/ubuntu/cephtest/url_file 2024-05-01T12:52:34.099 INFO:teuthology.orchestra.run.smithi050.stdout:http://smithi050.front.sepia.ceph.com:80 2024-05-01T12:52:34.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-05-01T12:52:34.164 INFO:tasks.rgw.client.0:Restarting daemon 2024-05-01T12:52:34.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-05-01T12:52:34.206 INFO:tasks.rgw.client.0:Started 2024-05-01T12:52:34.207 INFO:tasks.rgw:rgw client.1 config is {} 2024-05-01T12:52:34.207 INFO:tasks.rgw:Using beast as radosgw frontend 2024-05-01T12:52:34.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo echo -n http://smithi063.front.sepia.ceph.com:80 | sudo tee /home/ubuntu/cephtest/url_file 2024-05-01T12:52:34.233 INFO:teuthology.orchestra.run.smithi063.stdout:http://smithi063.front.sepia.ceph.com:80 2024-05-01T12:52:34.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-05-01T12:52:34.298 INFO:tasks.rgw.client.1:Restarting daemon 2024-05-01T12:52:34.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.1 --cluster ceph -k /etc/ceph/ceph.client.1.keyring --log-file /var/log/ceph/rgw.ceph.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.1.stdout 2>&1 2024-05-01T12:52:34.341 INFO:tasks.rgw.client.1:Started 2024-05-01T12:52:34.341 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi050.front.sepia.ceph.com:80/ 2024-05-01T12:52:34.341 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:80/ 2024-05-01T12:52:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-05-01T12:52:34.371 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-01T12:52:34.371 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2024-05-01T12:52:34.371 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-05-01T12:52:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:curl: (7) Failed to connect to smithi050.front.sepia.ceph.com port 80: Connection refused 2024-05-01T12:52:34.399 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T12:52:34.399+0000 7f130c997640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T12:52:34.399 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T12:52:34.399+0000 7f130c997640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T12:52:34.399 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T12:52:34.399+0000 7f130c997640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T12:52:34.399 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T12:52:34.399+0000 7f130c997640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T12:52:34.400 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T12:52:34.399+0000 7f1432434880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-05-01T12:52:34.557 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T12:52:34.557+0000 7f46d1952640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T12:52:34.557 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T12:52:34.557+0000 7f46d1952640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T12:52:34.557 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T12:52:34.557+0000 7f46d1952640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T12:52:34.558 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T12:52:34.557+0000 7f46d1952640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T12:52:34.558 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T12:52:34.557+0000 7f47f73ef880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-05-01T12:52:35.372 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:80/ 2024-05-01T12:52:35.394 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-01T12:52:35.394 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2024-05-01T12:52:35.396 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 93500 0 --:--:-- --:--:-- --:--:-- 93500 2024-05-01T12:52:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:anonymous 2024-05-01T12:52:35.397 INFO:tasks.rgw:Polling client.1 until it starts accepting connections on http://smithi063.front.sepia.ceph.com:80/ 2024-05-01T12:52:35.397 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:80/ 2024-05-01T12:52:35.431 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-05-01T12:52:35.431 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-05-01T12:52:35.434 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 62333 0 --:--:-- --:--:-- --:--:-- 93500 2024-05-01T12:52:35.434 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-05-01T12:52:35.434 INFO:teuthology.run_tasks:Running task notification-tests... 2024-05-01T12:52:35.445 DEBUG:tasks.notification_tests:Notifications config is {'client.0': {'rgw_server': 'client.0'}} 2024-05-01T12:52:35.446 INFO:tasks.notification_tests:Downloading bucket-notifications-tests... 2024-05-01T12:52:35.446 INFO:tasks.notification_tests:Using branch wip-alimasa-bn-test-testing from https://github.com/AliMasarweh/ceph.git for bucket notifications tests 2024-05-01T12:52:35.446 DEBUG:teuthology.orchestra.run.smithi050:> git clone -b wip-alimasa-bn-test-testing https://github.com/AliMasarweh/ceph.git /home/ubuntu/cephtest/ceph 2024-05-01T12:52:35.462 INFO:teuthology.orchestra.run.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/ceph'... 2024-05-01T12:53:11.878 INFO:tasks.notification_tests:Pre-processing... 2024-05-01T12:53:11.879 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p /home/ubuntu/.aws/models/s3/2006-03-01/ 2024-05-01T12:53:11.934 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/.aws/models/s3/2006-03-01/ && cp /home/ubuntu/cephtest/ceph/examples/rgw/boto3/service-2.sdk-extras.json service-2.sdk-extras.json 2024-05-01T12:53:11.989 INFO:tasks.notification_tests:Creating rgw user... 2024-05-01T12:53:11.990 DEBUG:tasks.notification_tests:Creating user foo.client.0 on client.0 2024-05-01T12:53:11.990 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key NRWJLBIBJRWYBMQBMFAB --secret h7fQvx2YI1qD7nEF75hp7k7KDtxh0jZGG7+OXVPXkUmTuVFICeV97w== --cluster ceph 2024-05-01T12:53:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:12.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.111+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.112+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:12.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.113+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.113+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.113+0000 7f5bf8a40800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.113+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.114+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.114+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.114+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:12.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.114+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.115+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.115+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:12.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:12.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.116+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 zone MyZone found 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.117+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.118+0000 7f5bf8a40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.118+0000 7f5bf8a40800 20 started sync module instance, tier type = 2024-05-01T12:53:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.118+0000 7f5bf8a40800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=0 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=1 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=2 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=3 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=4 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=5 2024-05-01T12:53:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.129+0000 7f5bf8a40800 20 add_watcher() i=6 2024-05-01T12:53:12.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.131+0000 7f5bf8a40800 20 add_watcher() i=7 2024-05-01T12:53:12.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.131+0000 7f5bf8a40800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:12.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5aa9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:12.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 5 note: GC not initialized 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5aa8fe1640 20 reqs_thread_entry: start 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 20 Filter name: none 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5a227fc640 20 reqs_thread_entry: start 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 10 cache get: name=MyZone.rgw.meta+users.uid+foo.client.0 : miss 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.133+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 adding MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 cache get: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB : miss 2024-05-01T12:53:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 20 rados->read ofs=0 len=0 2024-05-01T12:53:12.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:12.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 cache put: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB info.flags=0x0 2024-05-01T12:53:12.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 adding MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB to cache LRU end 2024-05-01T12:53:12.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 cache get: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB : hit (negative entry) 2024-05-01T12:53:12.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.134+0000 7f5bf8a40800 10 cache get: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB : hit (negative entry) 2024-05-01T12:53:12.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5bf8a40800 10 cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-05-01T12:53:12.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5bf8a40800 10 moving MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-05-01T12:53:12.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5bf8a40800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-05-01T12:53:12.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5ad6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855360 cookie 94202718138512 notifier 4779 bl.length()=627 2024-05-01T12:53:12.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5ad6ffd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-05-01T12:53:12.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.138+0000 7f5ad6ffd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5bf8a40800 10 cache put: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB info.flags=0x7 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5bf8a40800 10 moving MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB to cache LRU end 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5bf8a40800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:NRWJLBIBJRWYBMQBMFAB, ofs0, ns] 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5ad67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855360 cookie 94202718144144 notifier 4779 bl.length()=185 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5ad67fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB info.flags=0x7 2024-05-01T12:53:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.140+0000 7f5ad67fc640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+NRWJLBIBJRWYBMQBMFAB to cache LRU end 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "user_id": "foo.client.0", 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "display_name": "Mr. foo.client.0", 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "email": "", 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "suspended": 0, 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "max_buckets": 1000, 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "subusers": [], 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "keys": [ 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-05-01T12:53:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "user": "foo.client.0", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "NRWJLBIBJRWYBMQBMFAB", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "h7fQvx2YI1qD7nEF75hp7k7KDtxh0jZGG7+OXVPXkUmTuVFICeV97w==", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "active": true, 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "create_date": "2024-05-01T12:53:12.135541Z" 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "swift_keys": [], 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": [], 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "op_mask": "read, write, delete", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "default_placement": "", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "default_storage_class": "", 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_tags": [], 2024-05-01T12:53:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_quota": { 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "user_quota": { 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-05-01T12:53:12.144 INFO:teuthology.orchestra.run.smithi050.stdout: "temp_url_keys": [], 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "rgw", 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "mfa_ids": [], 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "account_id": "", 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "path": "/", 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "create_date": "2024-05-01T12:53:12.135527Z", 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "group_ids": [] 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-05-01T12:53:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.142+0000 7f5aa9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.144+0000 7f5bf8a40800 20 remove_watcher() i=0 2024-05-01T12:53:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.144+0000 7f5bf8a40800 2 removed watcher, disabling cache 2024-05-01T12:53:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.145+0000 7f5bf8a40800 20 remove_watcher() i=1 2024-05-01T12:53:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.146+0000 7f5bf8a40800 20 remove_watcher() i=2 2024-05-01T12:53:12.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.147+0000 7f5bf8a40800 20 remove_watcher() i=3 2024-05-01T12:53:12.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.148+0000 7f5bf8a40800 20 remove_watcher() i=4 2024-05-01T12:53:12.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.150+0000 7f5bf8a40800 20 remove_watcher() i=5 2024-05-01T12:53:12.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.151+0000 7f5bf8a40800 20 remove_watcher() i=6 2024-05-01T12:53:12.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:12.152+0000 7f5bf8a40800 20 remove_watcher() i=7 2024-05-01T12:53:12.163 INFO:tasks.notification_tests:Configuring bucket-notifications-tests... 2024-05-01T12:53:12.163 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-05-01T12:53:12.163 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf 2024-05-01T12:53:12.179 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification && ./bootstrap 2024-05-01T12:53:15.130 INFO:teuthology.orchestra.run.smithi050.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2024-05-01T12:53:15.659 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:03 ago on Wed 01 May 2024 12:49:12 PM UTC. 2024-05-01T12:53:15.791 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: libevent-devel x86_64 2.1.12-6.el9 appstream 99 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: libxml2-devel x86_64 2.9.13-5.el9 appstream 898 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: libxslt-devel x86_64 1.1.34-9.el9 appstream 320 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pika noarch 1.2.1-1.el9 epel 244 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-01T12:53:15.793 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: cmake-filesystem x86_64 3.26.5-2.el9 appstream 19 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: libgpg-error-devel x86_64 1.42-5.el9 appstream 66 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-Automat noarch 22.10.0-2.el9 epel 59 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-constantly noarch 15.1.0-25.el9 epel 20 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-hyperlink noarch 21.0.0-10.el9 epel 115 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-incremental noarch 21.3.0-5.el9 epel 34 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-service-identity noarch 21.1.0-5.el9 epel 27 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zope-interface x86_64 5.4.0-5.el9.1 epel 168 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: xz-devel x86_64 5.2.5-8.el9 appstream 55 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tornado x86_64 6.1.0-9.el9 appstream 698 k 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-twisted noarch 22.10.0-3.el9 epel 5.1 M 2024-05-01T12:53:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-twisted+tls noarch 22.10.0-3.el9 epel 9.4 k 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:Install 19 Packages 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 8.0 M 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 47 M 2024-05-01T12:53:15.795 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-05-01T12:53:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:(1/19): cmake-filesystem-3.26.5-2.el9.x86_64.rp 149 kB/s | 19 kB 00:00 2024-05-01T12:53:16.379 INFO:teuthology.orchestra.run.smithi050.stdout:(2/19): libgpg-error-devel-1.42-5.el9.x86_64.rp 282 kB/s | 66 kB 00:00 2024-05-01T12:53:16.405 INFO:teuthology.orchestra.run.smithi050.stdout:(3/19): libevent-devel-2.1.12-6.el9.x86_64.rpm 382 kB/s | 99 kB 00:00 2024-05-01T12:53:16.572 INFO:teuthology.orchestra.run.smithi050.stdout:(4/19): libxslt-devel-1.1.34-9.el9.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2024-05-01T12:53:16.672 INFO:teuthology.orchestra.run.smithi050.stdout:(5/19): libxml2-devel-2.9.13-5.el9.x86_64.rpm 2.2 MB/s | 898 kB 00:00 2024-05-01T12:53:16.722 INFO:teuthology.orchestra.run.smithi050.stdout:(6/19): xz-devel-5.2.5-8.el9.x86_64.rpm 1.1 MB/s | 55 kB 00:00 2024-05-01T12:53:16.756 INFO:teuthology.orchestra.run.smithi050.stdout:(7/19): python3-tornado-6.1.0-9.el9.x86_64.rpm 3.7 MB/s | 698 kB 00:00 2024-05-01T12:53:16.781 INFO:teuthology.orchestra.run.smithi050.stdout:(8/19): python3-attrs-20.3.0-7.el9.noarch.rpm 232 kB/s | 87 kB 00:00 2024-05-01T12:53:16.807 INFO:teuthology.orchestra.run.smithi050.stdout:(9/19): zlib-devel-1.2.11-41.el9.x86_64.rpm 533 kB/s | 45 kB 00:00 2024-05-01T12:53:16.857 INFO:teuthology.orchestra.run.smithi050.stdout:(10/19): python3-constantly-15.1.0-25.el9.noarc 263 kB/s | 20 kB 00:00 2024-05-01T12:53:16.882 INFO:teuthology.orchestra.run.smithi050.stdout:(11/19): python3-Automat-22.10.0-2.el9.noarch.r 469 kB/s | 59 kB 00:00 2024-05-01T12:53:16.908 INFO:teuthology.orchestra.run.smithi050.stdout:(12/19): python3-incremental-21.3.0-5.el9.noarc 683 kB/s | 34 kB 00:00 2024-05-01T12:53:16.933 INFO:teuthology.orchestra.run.smithi050.stdout:(13/19): python3-hyperlink-21.0.0-10.el9.noarch 914 kB/s | 115 kB 00:00 2024-05-01T12:53:16.958 INFO:teuthology.orchestra.run.smithi050.stdout:(14/19): python3-service-identity-21.1.0-5.el9. 531 kB/s | 27 kB 00:00 2024-05-01T12:53:16.983 INFO:teuthology.orchestra.run.smithi050.stdout:(15/19): python3-twisted+tls-22.10.0-3.el9.noar 189 kB/s | 9.4 kB 00:00 2024-05-01T12:53:17.009 INFO:teuthology.orchestra.run.smithi050.stdout:(16/19): python3-pika-1.2.1-1.el9.noarch.rpm 1.9 MB/s | 244 kB 00:00 2024-05-01T12:53:17.034 INFO:teuthology.orchestra.run.smithi050.stdout:(17/19): python3-xmltodict-0.12.0-15.el9.noarch 441 kB/s | 22 kB 00:00 2024-05-01T12:53:17.059 INFO:teuthology.orchestra.run.smithi050.stdout:(18/19): python3-zope-interface-5.4.0-5.el9.1.x 3.3 MB/s | 168 kB 00:00 2024-05-01T12:53:17.185 INFO:teuthology.orchestra.run.smithi050.stdout:(19/19): python3-twisted-22.10.0-3.el9.noarch.r 23 MB/s | 5.1 MB 00:00 2024-05-01T12:53:17.185 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-05-01T12:53:17.185 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.8 MB/s | 8.0 MB 00:01 2024-05-01T12:53:17.295 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-05-01T12:53:17.317 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-05-01T12:53:17.318 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-05-01T12:53:17.523 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-05-01T12:53:17.524 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-05-01T12:53:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-05-01T12:53:18.353 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 1/19 2024-05-01T12:53:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-Automat-22.10.0-2.el9.noarch 2/19 2024-05-01T12:53:18.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-service-identity-21.1.0-5.el9.noarch 3/19 2024-05-01T12:53:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zope-interface-5.4.0-5.el9.1.x86_64 4/19 2024-05-01T12:53:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-incremental-21.3.0-5.el9.noarch 5/19 2024-05-01T12:53:18.578 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-hyperlink-21.0.0-10.el9.noarch 6/19 2024-05-01T12:53:18.606 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-constantly-15.1.0-25.el9.noarch 7/19 2024-05-01T12:53:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-twisted+tls-22.10.0-3.el9.noarch 8/19 2024-05-01T12:53:19.767 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-twisted-22.10.0-3.el9.noarch 9/19 2024-05-01T12:53:19.799 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 10/19 2024-05-01T12:53:19.940 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xz-devel-5.2.5-8.el9.x86_64 11/19 2024-05-01T12:53:19.979 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tornado-6.1.0-9.el9.x86_64 12/19 2024-05-01T12:53:20.039 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgpg-error-devel-1.42-5.el9.x86_64 13/19 2024-05-01T12:53:20.248 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cmake-filesystem-3.26.5-2.el9.x86_64 14/19 2024-05-01T12:53:20.364 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libxml2-devel-2.9.13-5.el9.x86_64 15/19 2024-05-01T12:53:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libxslt-devel-1.1.34-9.el9.x86_64 16/19 2024-05-01T12:53:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pika-1.2.1-1.el9.noarch 17/19 2024-05-01T12:53:20.523 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 18/19 2024-05-01T12:53:20.590 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libevent-devel-2.1.12-6.el9.x86_64 19/19 2024-05-01T12:53:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libevent-devel-2.1.12-6.el9.x86_64 19/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cmake-filesystem-3.26.5-2.el9.x86_64 1/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libevent-devel-2.1.12-6.el9.x86_64 2/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgpg-error-devel-1.42-5.el9.x86_64 3/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libxml2-devel-2.9.13-5.el9.x86_64 4/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libxslt-devel-1.1.34-9.el9.x86_64 5/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 6/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tornado-6.1.0-9.el9.x86_64 7/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xz-devel-5.2.5-8.el9.x86_64 8/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 9/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-Automat-22.10.0-2.el9.noarch 10/19 2024-05-01T12:53:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-constantly-15.1.0-25.el9.noarch 11/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-hyperlink-21.0.0-10.el9.noarch 12/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-incremental-21.3.0-5.el9.noarch 13/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pika-1.2.1-1.el9.noarch 14/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-service-identity-21.1.0-5.el9.noarch 15/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-twisted+tls-22.10.0-3.el9.noarch 16/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-twisted-22.10.0-3.el9.noarch 17/19 2024-05-01T12:53:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 18/19 2024-05-01T12:53:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zope-interface-5.4.0-5.el9.1.x86_64 19/19 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: cmake-filesystem-3.26.5-2.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: libevent-devel-2.1.12-6.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: libgpg-error-devel-1.42-5.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: libxml2-devel-2.9.13-5.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: libxslt-devel-1.1.34-9.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-Automat-22.10.0-2.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-constantly-15.1.0-25.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-hyperlink-21.0.0-10.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-incremental-21.3.0-5.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pika-1.2.1-1.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-service-identity-21.1.0-5.el9.noarch 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tornado-6.1.0-9.el9.x86_64 2024-05-01T12:53:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-twisted-22.10.0-3.el9.noarch 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-twisted+tls-22.10.0-3.el9.noarch 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zope-interface-5.4.0-5.el9.1.x86_64 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: xz-devel-5.2.5-8.el9.x86_64 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:24.060 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-05-01T12:53:27.228 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.9/site-packages (21.3.1) 2024-05-01T12:53:27.477 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting pip 2024-05-01T12:53:27.572 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading pip-24.0-py3-none-any.whl (2.1 MB) 2024-05-01T12:53:28.010 INFO:teuthology.orchestra.run.smithi050.stdout:Installing collected packages: pip 2024-05-01T12:53:28.010 INFO:teuthology.orchestra.run.smithi050.stdout: Attempting uninstall: pip 2024-05-01T12:53:28.011 INFO:teuthology.orchestra.run.smithi050.stdout: Found existing installation: pip 21.3.1 2024-05-01T12:53:28.206 INFO:teuthology.orchestra.run.smithi050.stdout: Uninstalling pip-21.3.1: 2024-05-01T12:53:28.212 INFO:teuthology.orchestra.run.smithi050.stdout: Successfully uninstalled pip-21.3.1 2024-05-01T12:53:29.265 INFO:teuthology.orchestra.run.smithi050.stdout:Successfully installed pip-24.0 2024-05-01T12:53:30.059 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting nose>=1.0.0 (from -r requirements.txt (line 1)) 2024-05-01T12:53:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading nose-1.3.7-py3-none-any.whl.metadata (1.7 kB) 2024-05-01T12:53:30.299 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting boto>=2.6.0 (from -r requirements.txt (line 2)) 2024-05-01T12:53:30.317 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl.metadata (7.3 kB) 2024-05-01T12:53:30.985 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 3)) 2024-05-01T12:53:31.001 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading boto3-1.34.95-py3-none-any.whl.metadata (6.6 kB) 2024-05-01T12:53:31.127 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting configparser>=5.0.0 (from -r requirements.txt (line 4)) 2024-05-01T12:53:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading configparser-7.0.0-py3-none-any.whl.metadata (5.4 kB) 2024-05-01T12:53:31.277 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting kafka-python>=2.0.0 (from -r requirements.txt (line 5)) 2024-05-01T12:53:31.295 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading kafka_python-2.0.2-py2.py3-none-any.whl.metadata (7.8 kB) 2024-05-01T12:53:31.347 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: pika in /usr/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (1.2.1) 2024-05-01T12:53:31.429 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting pika (from -r requirements.txt (line 6)) 2024-05-01T12:53:31.447 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading pika-1.3.2-py3-none-any.whl.metadata (13 kB) 2024-05-01T12:53:31.586 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting cloudevents (from -r requirements.txt (line 7)) 2024-05-01T12:53:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading cloudevents-1.10.1-py3-none-any.whl.metadata (6.9 kB) 2024-05-01T12:53:31.658 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: xmltodict in /usr/lib/python3.9/site-packages (from -r requirements.txt (line 8)) (0.12.0) 2024-05-01T12:53:31.743 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting xmltodict (from -r requirements.txt (line 8)) 2024-05-01T12:53:31.779 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB) 2024-05-01T12:53:32.546 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting botocore<1.35.0,>=1.34.95 (from boto3>=1.0.0->-r requirements.txt (line 3)) 2024-05-01T12:53:32.586 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading botocore-1.34.95-py3-none-any.whl.metadata (5.7 kB) 2024-05-01T12:53:32.739 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3>=1.0.0->-r requirements.txt (line 3)) 2024-05-01T12:53:32.774 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB) 2024-05-01T12:53:32.945 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting s3transfer<0.11.0,>=0.10.0 (from boto3>=1.0.0->-r requirements.txt (line 3)) 2024-05-01T12:53:32.983 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB) 2024-05-01T12:53:33.291 INFO:teuthology.orchestra.run.smithi050.stdout:Collecting deprecation<3.0,>=2.0 (from cloudevents->-r requirements.txt (line 7)) 2024-05-01T12:53:33.328 INFO:teuthology.orchestra.run.smithi050.stdout: Downloading deprecation-2.1.0-py2.py3-none-any.whl.metadata (4.6 kB) 2024-05-01T12:53:33.372 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.95->boto3>=1.0.0->-r requirements.txt (line 3)) (2.8.1) 2024-05-01T12:53:33.373 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.95->boto3>=1.0.0->-r requirements.txt (line 3)) (1.26.5) 2024-05-01T12:53:33.375 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: packaging in /usr/lib/python3.9/site-packages (from deprecation<3.0,>=2.0->cloudevents->-r requirements.txt (line 7)) (20.9) 2024-05-01T12:53:33.556 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.95->boto3>=1.0.0->-r requirements.txt (line 3)) (1.15.0) 2024-05-01T12:53:33.563 INFO:teuthology.orchestra.run.smithi050.stdout:Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3.9/site-packages (from packaging->deprecation<3.0,>=2.0->cloudevents->-r requirements.txt (line 7)) (2.4.7) 2024-05-01T12:53:33.608 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2024-05-01T12:53:33.757 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.7/154.7 kB 1.0 MB/s eta 0:00:00 2024-05-01T12:53:33.797 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2024-05-01T12:53:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 2.1 MB/s eta 0:00:00 2024-05-01T12:53:34.476 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading boto3-1.34.95-py3-none-any.whl (139 kB) 2024-05-01T12:53:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 1.4 MB/s eta 0:00:00 2024-05-01T12:53:34.624 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading configparser-7.0.0-py3-none-any.whl (16 kB) 2024-05-01T12:53:34.729 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading kafka_python-2.0.2-py2.py3-none-any.whl (246 kB) 2024-05-01T12:53:34.860 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 246.5/246.5 kB 1.9 MB/s eta 0:00:00 2024-05-01T12:53:34.914 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading pika-1.3.2-py3-none-any.whl (155 kB) 2024-05-01T12:53:35.010 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.4/155.4 kB 1.6 MB/s eta 0:00:00 2024-05-01T12:53:35.054 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading cloudevents-1.10.1-py3-none-any.whl (55 kB) 2024-05-01T12:53:35.119 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.0/55.0 kB 736.5 kB/s eta 0:00:00 2024-05-01T12:53:35.161 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB) 2024-05-01T12:53:35.246 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading botocore-1.34.95-py3-none-any.whl (12.2 MB) 2024-05-01T12:53:37.915 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 4.8 MB/s eta 0:00:00 2024-05-01T12:53:37.941 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB) 2024-05-01T12:53:38.012 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2024-05-01T12:53:38.088 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading s3transfer-0.10.1-py3-none-any.whl (82 kB) 2024-05-01T12:53:38.139 INFO:teuthology.orchestra.run.smithi050.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.2/82.2 kB 1.5 MB/s eta 0:00:00 2024-05-01T12:53:38.564 INFO:teuthology.orchestra.run.smithi050.stdout:Installing collected packages: nose, kafka-python, boto, xmltodict, pika, jmespath, configparser, deprecation, botocore, s3transfer, cloudevents, boto3 2024-05-01T12:53:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: Attempting uninstall: xmltodict 2024-05-01T12:53:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: Found existing installation: xmltodict 0.12.0 2024-05-01T12:53:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: Not uninstalling xmltodict at /usr/lib/python3.9/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2024-05-01T12:53:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: Can't uninstall 'xmltodict'. No files were found to uninstall. 2024-05-01T12:53:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: Attempting uninstall: pika 2024-05-01T12:53:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: Found existing installation: pika 1.2.1 2024-05-01T12:53:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: Not uninstalling pika at /usr/lib/python3.9/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2024-05-01T12:53:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: Can't uninstall 'pika'. No files were found to uninstall. 2024-05-01T12:53:40.683 INFO:teuthology.orchestra.run.smithi050.stdout:Successfully installed boto-2.49.0 boto3-1.34.95 botocore-1.34.95 cloudevents-1.10.1 configparser-7.0.0 deprecation-2.1.0 jmespath-1.0.1 kafka-python-2.0.2 nose-1.3.7 pika-1.3.2 s3transfer-0.10.1 xmltodict-0.13.0 2024-05-01T12:53:41.051 INFO:teuthology.orchestra.run.smithi050.stdout:running develop 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:running egg_info 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:creating bn_tests.egg-info 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:writing bn_tests.egg-info/PKG-INFO 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:writing dependency_links to bn_tests.egg-info/dependency_links.txt 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:writing requirements to bn_tests.egg-info/requires.txt 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:writing top-level names to bn_tests.egg-info/top_level.txt 2024-05-01T12:53:41.135 INFO:teuthology.orchestra.run.smithi050.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-05-01T12:53:41.136 INFO:teuthology.orchestra.run.smithi050.stdout:reading manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-05-01T12:53:41.136 INFO:teuthology.orchestra.run.smithi050.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-05-01T12:53:41.141 INFO:teuthology.orchestra.run.smithi050.stdout:running build_ext 2024-05-01T12:53:41.150 INFO:teuthology.orchestra.run.smithi050.stdout:Creating /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages/bn-tests.egg-link (link to .) 2024-05-01T12:53:41.150 INFO:teuthology.orchestra.run.smithi050.stdout:Adding bn-tests 0.0.1 to easy-install.pth file 2024-05-01T12:53:41.151 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.151 INFO:teuthology.orchestra.run.smithi050.stdout:Installed /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification 2024-05-01T12:53:41.152 INFO:teuthology.orchestra.run.smithi050.stdout:Processing dependencies for bn-tests==0.0.1 2024-05-01T12:53:41.166 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for boto3==1.34.95 2024-05-01T12:53:41.166 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: boto3 1.34.95 2024-05-01T12:53:41.166 INFO:teuthology.orchestra.run.smithi050.stdout:Adding boto3 1.34.95 to easy-install.pth file 2024-05-01T12:53:41.166 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.166 INFO:teuthology.orchestra.run.smithi050.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for boto==2.49.0 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: boto 2.49.0 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Adding boto 2.49.0 to easy-install.pth file 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for jmespath==1.0.1 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: jmespath 1.0.1 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Adding jmespath 1.0.1 to easy-install.pth file 2024-05-01T12:53:41.169 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.170 INFO:teuthology.orchestra.run.smithi050.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2024-05-01T12:53:41.170 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for botocore==1.34.95 2024-05-01T12:53:41.170 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: botocore 1.34.95 2024-05-01T12:53:41.171 INFO:teuthology.orchestra.run.smithi050.stdout:Adding botocore 1.34.95 to easy-install.pth file 2024-05-01T12:53:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.171 INFO:teuthology.orchestra.run.smithi050.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2024-05-01T12:53:41.171 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for s3transfer==0.10.1 2024-05-01T12:53:41.171 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: s3transfer 0.10.1 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Adding s3transfer 0.10.1 to easy-install.pth file 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.9/site-packages 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for python-dateutil==2.8.1 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: python-dateutil 2.8.1 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Adding python-dateutil 2.8.1 to easy-install.pth file 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.172 INFO:teuthology.orchestra.run.smithi050.stdout:Using /usr/lib/python3.9/site-packages 2024-05-01T12:53:41.188 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for urllib3==1.26.5 2024-05-01T12:53:41.189 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: urllib3 1.26.5 2024-05-01T12:53:41.189 INFO:teuthology.orchestra.run.smithi050.stdout:Adding urllib3 1.26.5 to easy-install.pth file 2024-05-01T12:53:41.189 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.189 INFO:teuthology.orchestra.run.smithi050.stdout:Using /usr/lib/python3.9/site-packages 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Searching for six==1.15.0 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Best match: six 1.15.0 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Adding six 1.15.0 to easy-install.pth file 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Using /usr/lib/python3.9/site-packages 2024-05-01T12:53:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Finished processing dependencies for bn-tests==0.0.1 2024-05-01T12:53:41.210 INFO:tasks.notification_tests:Running bucket-notifications-tests... 2024-05-01T12:53:41.210 DEBUG:teuthology.orchestra.run.smithi050:bucket notification tests against different endpoints> BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!kafka_security_test,!modification_required,!manual_test' 2024-05-01T12:53:41.687 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 topics set/get/delete on master ... ignoring --setuser ceph since I am not root 2024-05-01T12:53:41.687 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.724+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.726+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.726+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.727+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.727+0000 7f8a9692d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.727+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.727+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.727+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.728+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.728+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.729+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.730+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.730+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 zone MyZone found 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 started sync module instance, tier type = 2024-05-01T12:53:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.731+0000 7f8a9692d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=0 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=1 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=2 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=3 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=4 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.745+0000 7f8a9692d800 20 add_watcher() i=5 2024-05-01T12:53:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.746+0000 7f8a9692d800 20 add_watcher() i=6 2024-05-01T12:53:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.748+0000 7f8a9692d800 20 add_watcher() i=7 2024-05-01T12:53:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.748+0000 7f8a9692d800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8948fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 5 note: GC not initialized 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f88c37fe640 20 reqs_thread_entry: start 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 20 Filter name: none 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f88c17fa640 20 reqs_thread_entry: start 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 : miss 2024-05-01T12:53:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.751+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 info.flags=0x0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 adding MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 to cache LRU end 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568021.643352 : miss 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568021.643352 info.flags=0x0 2024-05-01T12:53:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 adding MyZone.rgw.meta+users.keys+1714568021.643352 to cache LRU end 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568021.643352 : hit (negative entry) 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.752+0000 7f8a9692d800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568021.643352 : hit (negative entry) 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8a9692d800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 info.flags=0x17 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8a9692d800 10 moving MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 to cache LRU end 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8a9692d800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom$superman1714568021.6433594, ofs0, ns] 2024-05-01T12:53:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8975ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855361 cookie 93949288187728 notifier 4619 bl.length()=597 2024-05-01T12:53:41.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8975ffb640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 info.flags=0x17 2024-05-01T12:53:41.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.754+0000 7f8975ffb640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom$superman1714568021.6433594 to cache LRU end 2024-05-01T12:53:41.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f8a9692d800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568021.643352 info.flags=0x7 2024-05-01T12:53:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f8a9692d800 10 moving MyZone.rgw.meta+users.keys+1714568021.643352 to cache LRU end 2024-05-01T12:53:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f8a9692d800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568021.643352, ofs0, ns] 2024-05-01T12:53:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f89757fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855360 cookie 93949288213392 notifier 4619 bl.length()=203 2024-05-01T12:53:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f89757fa640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568021.643352 info.flags=0x7 2024-05-01T12:53:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.756+0000 7f89757fa640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568021.643352 to cache LRU end 2024-05-01T12:53:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.758+0000 7f8948fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:41.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.760+0000 7f8a9692d800 20 remove_watcher() i=0 2024-05-01T12:53:41.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.760+0000 7f8a9692d800 2 removed watcher, disabling cache 2024-05-01T12:53:41.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.761+0000 7f8a9692d800 20 remove_watcher() i=1 2024-05-01T12:53:41.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.762+0000 7f8a9692d800 20 remove_watcher() i=2 2024-05-01T12:53:41.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.763+0000 7f8a9692d800 20 remove_watcher() i=3 2024-05-01T12:53:41.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.765+0000 7f8a9692d800 20 remove_watcher() i=4 2024-05-01T12:53:41.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.766+0000 7f8a9692d800 20 remove_watcher() i=5 2024-05-01T12:53:41.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.767+0000 7f8a9692d800 20 remove_watcher() i=6 2024-05-01T12:53:41.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.768+0000 7f8a9692d800 20 remove_watcher() i=7 2024-05-01T12:53:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.853+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.854+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.854+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.855+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.855+0000 7f7352e10800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.855+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.855+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.855+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.856+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.856+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.857+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.858+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.858+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 zone MyZone found 2024-05-01T12:53:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.859+0000 7f7352e10800 20 rados->read ofs=0 len=0 2024-05-01T12:53:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.860+0000 7f7352e10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.860+0000 7f7352e10800 20 started sync module instance, tier type = 2024-05-01T12:53:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.860+0000 7f7352e10800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=0 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=1 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=2 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=3 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=4 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=5 2024-05-01T12:53:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.873+0000 7f7352e10800 20 add_watcher() i=6 2024-05-01T12:53:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.874+0000 7f7352e10800 20 add_watcher() i=7 2024-05-01T12:53:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.874+0000 7f7352e10800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.876+0000 7f7204fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.876+0000 7f7352e10800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.876+0000 7f7352e10800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.876+0000 7f7352e10800 5 note: GC not initialized 2024-05-01T12:53:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.877+0000 7f717f7fe640 20 reqs_thread_entry: start 2024-05-01T12:53:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.877+0000 7f7352e10800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.877+0000 7f7352e10800 20 Filter name: none 2024-05-01T12:53:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.877+0000 7f717d7fa640 20 reqs_thread_entry: start 2024-05-01T12:53:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.888+0000 7f7204fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.890+0000 7f7352e10800 20 remove_watcher() i=0 2024-05-01T12:53:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.890+0000 7f7352e10800 2 removed watcher, disabling cache 2024-05-01T12:53:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.891+0000 7f7352e10800 20 remove_watcher() i=1 2024-05-01T12:53:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.892+0000 7f7352e10800 20 remove_watcher() i=2 2024-05-01T12:53:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.894+0000 7f7352e10800 20 remove_watcher() i=3 2024-05-01T12:53:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.895+0000 7f7352e10800 20 remove_watcher() i=4 2024-05-01T12:53:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.896+0000 7f7352e10800 20 remove_watcher() i=5 2024-05-01T12:53:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.897+0000 7f7352e10800 20 remove_watcher() i=6 2024-05-01T12:53:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:41.898+0000 7f7352e10800 20 remove_watcher() i=7 2024-05-01T12:53:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:53:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 topics set/get/delete on master ... ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.159+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.160+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.160+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.161+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.161+0000 7f99f1936800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.161+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.161+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.161+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.162+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.162+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.163+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.163+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.163+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.163+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.164+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.164+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.164+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.164+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 zone MyZone found 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.165+0000 7f99f1936800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.166+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.166+0000 7f99f1936800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.166+0000 7f99f1936800 20 started sync module instance, tier type = 2024-05-01T12:53:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.166+0000 7f99f1936800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=0 2024-05-01T12:53:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=1 2024-05-01T12:53:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=2 2024-05-01T12:53:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=3 2024-05-01T12:53:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=4 2024-05-01T12:53:42.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=5 2024-05-01T12:53:42.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.180+0000 7f99f1936800 20 add_watcher() i=6 2024-05-01T12:53:42.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.181+0000 7f99f1936800 20 add_watcher() i=7 2024-05-01T12:53:42.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.181+0000 7f99f1936800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.183+0000 7f98a37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 5 note: GC not initialized 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f98a27e4640 20 reqs_thread_entry: start 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 Filter name: none 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f9813fff640 20 reqs_thread_entry: start 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 : miss 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 info.flags=0x0 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 10 adding MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 to cache LRU end 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568022.0783074 : miss 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.184+0000 7f99f1936800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.185+0000 7f99f1936800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.185+0000 7f99f1936800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568022.0783074 info.flags=0x0 2024-05-01T12:53:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.185+0000 7f99f1936800 10 adding MyZone.rgw.meta+users.keys+1714568022.0783074 to cache LRU end 2024-05-01T12:53:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.185+0000 7f99f1936800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568022.0783074 : hit (negative entry) 2024-05-01T12:53:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.185+0000 7f99f1936800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568022.0783074 : hit (negative entry) 2024-05-01T12:53:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f99f1936800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 info.flags=0x17 2024-05-01T12:53:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f99f1936800 10 moving MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 to cache LRU end 2024-05-01T12:53:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f99f1936800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom$superman1714568022.0783145, ofs0, ns] 2024-05-01T12:53:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f98d0ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855362 cookie 94504105067328 notifier 4839 bl.length()=598 2024-05-01T12:53:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f98d0ff9640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 info.flags=0x17 2024-05-01T12:53:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.186+0000 7f98d0ff9640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom$superman1714568022.0783145 to cache LRU end 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f99f1936800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568022.0783074 info.flags=0x7 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f99f1936800 10 moving MyZone.rgw.meta+users.keys+1714568022.0783074 to cache LRU end 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f99f1936800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568022.0783074, ofs0, ns] 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f98d0ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855361 cookie 94504105058112 notifier 4839 bl.length()=204 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f98d0ff9640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568022.0783074 info.flags=0x7 2024-05-01T12:53:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.188+0000 7f98d0ff9640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568022.0783074 to cache LRU end 2024-05-01T12:53:42.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.190+0000 7f98a37e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.192+0000 7f99f1936800 20 remove_watcher() i=0 2024-05-01T12:53:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.192+0000 7f99f1936800 2 removed watcher, disabling cache 2024-05-01T12:53:42.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.193+0000 7f99f1936800 20 remove_watcher() i=1 2024-05-01T12:53:42.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.194+0000 7f99f1936800 20 remove_watcher() i=2 2024-05-01T12:53:42.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.195+0000 7f99f1936800 20 remove_watcher() i=3 2024-05-01T12:53:42.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.196+0000 7f99f1936800 20 remove_watcher() i=4 2024-05-01T12:53:42.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.197+0000 7f99f1936800 20 remove_watcher() i=5 2024-05-01T12:53:42.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.199+0000 7f99f1936800 20 remove_watcher() i=6 2024-05-01T12:53:42.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.200+0000 7f99f1936800 20 remove_watcher() i=7 2024-05-01T12:53:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.286+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.288+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.288+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.288+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.288+0000 7f61d2408800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.288+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.289+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.289+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.289+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.289+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.290+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.290+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.291+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.291+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.291+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.291+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 zone MyZone found 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.292+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.293+0000 7f61d2408800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.293+0000 7f61d2408800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.293+0000 7f61d2408800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.293+0000 7f61d2408800 20 started sync module instance, tier type = 2024-05-01T12:53:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.293+0000 7f61d2408800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=0 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=1 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=2 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=3 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=4 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.303+0000 7f61d2408800 20 add_watcher() i=5 2024-05-01T12:53:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.304+0000 7f61d2408800 20 add_watcher() i=6 2024-05-01T12:53:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.304+0000 7f61d2408800 20 add_watcher() i=7 2024-05-01T12:53:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.304+0000 7f61d2408800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.306+0000 7f6085fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f61d2408800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f61d2408800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f61d2408800 5 note: GC not initialized 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f6084fe1640 20 reqs_thread_entry: start 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f61d2408800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f61d2408800 20 Filter name: none 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.307+0000 7f5ffe7fc640 20 reqs_thread_entry: start 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.318+0000 7f6085fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.320+0000 7f61d2408800 20 remove_watcher() i=0 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.320+0000 7f61d2408800 2 removed watcher, disabling cache 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.321+0000 7f61d2408800 20 remove_watcher() i=1 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.322+0000 7f61d2408800 20 remove_watcher() i=2 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.323+0000 7f61d2408800 20 remove_watcher() i=3 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.324+0000 7f61d2408800 20 remove_watcher() i=4 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.325+0000 7f61d2408800 20 remove_watcher() i=5 2024-05-01T12:53:42.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.327+0000 7f61d2408800 20 remove_watcher() i=6 2024-05-01T12:53:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.328+0000 7f61d2408800 20 remove_watcher() i=7 2024-05-01T12:53:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.445+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.447+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.447+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.448+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.448+0000 7f3639813800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.448+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.448+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.448+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.449+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.449+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.450+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.451+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.451+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 zone MyZone found 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 started sync module instance, tier type = 2024-05-01T12:53:42.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.452+0000 7f3639813800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=0 2024-05-01T12:53:42.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=1 2024-05-01T12:53:42.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=2 2024-05-01T12:53:42.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=3 2024-05-01T12:53:42.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=4 2024-05-01T12:53:42.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=5 2024-05-01T12:53:42.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.463+0000 7f3639813800 20 add_watcher() i=6 2024-05-01T12:53:42.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.464+0000 7f3639813800 20 add_watcher() i=7 2024-05-01T12:53:42.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.464+0000 7f3639813800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.466+0000 7f34eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.466+0000 7f3639813800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.466+0000 7f3639813800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.466+0000 7f3639813800 5 note: GC not initialized 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.467+0000 7f34e9fe3640 20 reqs_thread_entry: start 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.467+0000 7f3639813800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.467+0000 7f3639813800 20 Filter name: none 2024-05-01T12:53:42.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.467+0000 7f345b7fe640 20 reqs_thread_entry: start 2024-05-01T12:53:42.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.467+0000 7f3639813800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.468+0000 7f3639813800 20 rados_obj.operate() r=0 bl.length=234 2024-05-01T12:53:42.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.469+0000 7f34eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.472+0000 7f3639813800 20 remove_watcher() i=0 2024-05-01T12:53:42.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.472+0000 7f3639813800 2 removed watcher, disabling cache 2024-05-01T12:53:42.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.472+0000 7f3639813800 20 remove_watcher() i=1 2024-05-01T12:53:42.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.474+0000 7f3639813800 20 remove_watcher() i=2 2024-05-01T12:53:42.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.474+0000 7f3639813800 20 remove_watcher() i=3 2024-05-01T12:53:42.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.475+0000 7f3639813800 20 remove_watcher() i=4 2024-05-01T12:53:42.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.477+0000 7f3639813800 20 remove_watcher() i=5 2024-05-01T12:53:42.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.478+0000 7f3639813800 20 remove_watcher() i=6 2024-05-01T12:53:42.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.480+0000 7f3639813800 20 remove_watcher() i=7 2024-05-01T12:53:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.566+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.567+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.567+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.568+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.568+0000 7fb216210800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.568+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.568+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.568+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.569+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.569+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.570+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.570+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.570+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.570+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.571+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.571+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.571+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.571+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 zone MyZone found 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.572+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.573+0000 7fb216210800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.573+0000 7fb216210800 20 started sync module instance, tier type = 2024-05-01T12:53:42.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.573+0000 7fb216210800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=0 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=1 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=2 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=3 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=4 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=5 2024-05-01T12:53:42.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.587+0000 7fb216210800 20 add_watcher() i=6 2024-05-01T12:53:42.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.588+0000 7fb216210800 20 add_watcher() i=7 2024-05-01T12:53:42.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.588+0000 7fb216210800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb0bb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 5 note: GC not initialized 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb0ba7fc640 20 reqs_thread_entry: start 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 20 Filter name: none 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb033fff640 20 reqs_thread_entry: start 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.590+0000 7fb216210800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : miss 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom info.flags=0x0 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 adding MyZone.rgw.log++pubsub.kaboom to cache LRU end 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : hit (negative entry) 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_3 : miss 2024-05-01T12:53:42.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 20 rados_obj.operate() r=0 bl.length=234 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_3 info.flags=0x15 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 adding MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_3 to cache LRU end 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.591+0000 7fb216210800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_3 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.592+0000 7fb216210800 10 removing MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_3 from cache 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.592+0000 7fb216210800 10 distributing notification oid=MyZone.rgw.control:notify.6 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.kaboom:ashubx-2_topic_3, ofs0, ns] 2024-05-01T12:53:42.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.593+0000 7fb0f57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855370 cookie 94359700041200 notifier 4896 bl.length()=174 2024-05-01T12:53:42.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.594+0000 7fb216210800 20 WARNING: failed to remove topic buckets obj buckets.kaboom:ashubx-2_topic_3 with: (2) No such file or directory 2024-05-01T12:53:42.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.595+0000 7fb0bb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.597+0000 7fb216210800 20 remove_watcher() i=0 2024-05-01T12:53:42.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.597+0000 7fb216210800 2 removed watcher, disabling cache 2024-05-01T12:53:42.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.598+0000 7fb216210800 20 remove_watcher() i=1 2024-05-01T12:53:42.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.599+0000 7fb216210800 20 remove_watcher() i=2 2024-05-01T12:53:42.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.600+0000 7fb216210800 20 remove_watcher() i=3 2024-05-01T12:53:42.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.601+0000 7fb216210800 20 remove_watcher() i=4 2024-05-01T12:53:42.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.602+0000 7fb216210800 20 remove_watcher() i=5 2024-05-01T12:53:42.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.603+0000 7fb216210800 20 remove_watcher() i=6 2024-05-01T12:53:42.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.605+0000 7fb216210800 20 remove_watcher() i=7 2024-05-01T12:53:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.690+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.692+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.692+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.692+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.692+0000 7f7d2580e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.692+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.693+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.693+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.693+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.693+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.694+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.694+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.694+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.694+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.695+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.695+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.695+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.695+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 zone MyZone found 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.696+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.697+0000 7f7d2580e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.697+0000 7f7d2580e800 20 started sync module instance, tier type = 2024-05-01T12:53:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.697+0000 7f7d2580e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=0 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=1 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=2 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=3 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=4 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.709+0000 7f7d2580e800 20 add_watcher() i=5 2024-05-01T12:53:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.710+0000 7f7d2580e800 20 add_watcher() i=6 2024-05-01T12:53:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.712+0000 7f7d2580e800 20 add_watcher() i=7 2024-05-01T12:53:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.712+0000 7f7d2580e800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7bd9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7d2580e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7d2580e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7d2580e800 5 note: GC not initialized 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7bd8fe1640 20 reqs_thread_entry: start 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7d2580e800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7d2580e800 20 Filter name: none 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.714+0000 7f7b527fc640 20 reqs_thread_entry: start 2024-05-01T12:53:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.715+0000 7f7d2580e800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:ERROR: could not get topic: 2024-05-01T12:53:42.715+0000 7f7d2580e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.715+0000 7f7d2580e800 1 failed to read topic info for name: ashubx-2_topic_3 tenant: kaboom, ret=-2 2024-05-01T12:53:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:(2) No such file or directory 2024-05-01T12:53:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.716+0000 7f7bd9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.718+0000 7f7d2580e800 20 remove_watcher() i=0 2024-05-01T12:53:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.718+0000 7f7d2580e800 2 removed watcher, disabling cache 2024-05-01T12:53:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.719+0000 7f7d2580e800 20 remove_watcher() i=1 2024-05-01T12:53:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.720+0000 7f7d2580e800 20 remove_watcher() i=2 2024-05-01T12:53:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.721+0000 7f7d2580e800 20 remove_watcher() i=3 2024-05-01T12:53:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.722+0000 7f7d2580e800 20 remove_watcher() i=4 2024-05-01T12:53:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.723+0000 7f7d2580e800 20 remove_watcher() i=5 2024-05-01T12:53:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.724+0000 7f7d2580e800 20 remove_watcher() i=6 2024-05-01T12:53:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.725+0000 7f7d2580e800 20 remove_watcher() i=7 2024-05-01T12:53:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.808+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.810+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.810+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.811+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.811+0000 7fe680525800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.811+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.811+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.811+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.812+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.812+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.812+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.812+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.813+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.813+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.813+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.813+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 zone MyZone found 2024-05-01T12:53:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.814+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.815+0000 7fe680525800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.815+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.815+0000 7fe680525800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.815+0000 7fe680525800 20 started sync module instance, tier type = 2024-05-01T12:53:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.815+0000 7fe680525800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=0 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=1 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=2 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=3 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=4 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=5 2024-05-01T12:53:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.826+0000 7fe680525800 20 add_watcher() i=6 2024-05-01T12:53:42.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.827+0000 7fe680525800 20 add_watcher() i=7 2024-05-01T12:53:42.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.827+0000 7fe680525800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.828+0000 7fe532fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe680525800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe680525800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe680525800 5 note: GC not initialized 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe531fdb640 20 reqs_thread_entry: start 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe680525800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe680525800 20 Filter name: none 2024-05-01T12:53:42.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.829+0000 7fe4a37fe640 20 reqs_thread_entry: start 2024-05-01T12:53:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.838+0000 7fe680525800 20 RGWRados::pool_iterate: got topic.kaboom:ashubx-2_topic_2 2024-05-01T12:53:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.838+0000 7fe680525800 20 RGWRados::pool_iterate: got topic.kaboom:ashubx-2_topic_1 2024-05-01T12:53:42.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.838+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.839+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=234 2024-05-01T12:53:42.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.839+0000 7fe680525800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.839+0000 7fe680525800 20 rados_obj.operate() r=0 bl.length=288 2024-05-01T12:53:42.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.842+0000 7fe532fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.844+0000 7fe680525800 20 remove_watcher() i=0 2024-05-01T12:53:42.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.844+0000 7fe680525800 2 removed watcher, disabling cache 2024-05-01T12:53:42.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.845+0000 7fe680525800 20 remove_watcher() i=1 2024-05-01T12:53:42.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.846+0000 7fe680525800 20 remove_watcher() i=2 2024-05-01T12:53:42.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.847+0000 7fe680525800 20 remove_watcher() i=3 2024-05-01T12:53:42.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.848+0000 7fe680525800 20 remove_watcher() i=4 2024-05-01T12:53:42.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.849+0000 7fe680525800 20 remove_watcher() i=5 2024-05-01T12:53:42.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.850+0000 7fe680525800 20 remove_watcher() i=6 2024-05-01T12:53:42.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.851+0000 7fe680525800 20 remove_watcher() i=7 2024-05-01T12:53:42.901 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:42.901 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.937+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.938+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.938+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.939+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.939+0000 7f411552b800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.939+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.939+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.939+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.940+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:42.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.940+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.941+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:42.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.941+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.942+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.942+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.942+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:42.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.942+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:42.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 zone MyZone found 2024-05-01T12:53:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.943+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.944+0000 7f411552b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.944+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.944+0000 7f411552b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.944+0000 7f411552b800 20 started sync module instance, tier type = 2024-05-01T12:53:42.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.944+0000 7f411552b800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=0 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=1 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=2 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=3 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=4 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=5 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.956+0000 7f411552b800 20 add_watcher() i=6 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.957+0000 7f411552b800 20 add_watcher() i=7 2024-05-01T12:53:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.957+0000 7f411552b800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.958+0000 7f3fc77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 5 note: GC not initialized 2024-05-01T12:53:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f3fc67e4640 20 reqs_thread_entry: start 2024-05-01T12:53:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 20 Filter name: none 2024-05-01T12:53:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f3f37fff640 20 reqs_thread_entry: start 2024-05-01T12:53:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : miss 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom info.flags=0x0 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 10 adding MyZone.rgw.log++pubsub.kaboom to cache LRU end 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : hit (negative entry) 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_1 : miss 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.959+0000 7f411552b800 20 rados->read ofs=0 len=0 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.960+0000 7f411552b800 20 rados_obj.operate() r=0 bl.length=288 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.960+0000 7f411552b800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_1 info.flags=0x15 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.960+0000 7f411552b800 10 adding MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_1 to cache LRU end 2024-05-01T12:53:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.960+0000 7f411552b800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_1 2024-05-01T12:53:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.961+0000 7f411552b800 10 removing MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_1 from cache 2024-05-01T12:53:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.961+0000 7f411552b800 10 distributing notification oid=MyZone.rgw.control:notify.6 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.kaboom:ashubx-2_topic_1, ofs0, ns] 2024-05-01T12:53:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.961+0000 7f3ff4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855371 cookie 93901089317168 notifier 4956 bl.length()=174 2024-05-01T12:53:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.963+0000 7f411552b800 20 WARNING: failed to remove topic buckets obj buckets.kaboom:ashubx-2_topic_1 with: (2) No such file or directory 2024-05-01T12:53:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.964+0000 7f3fc77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.966+0000 7f411552b800 20 remove_watcher() i=0 2024-05-01T12:53:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.966+0000 7f411552b800 2 removed watcher, disabling cache 2024-05-01T12:53:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.967+0000 7f411552b800 20 remove_watcher() i=1 2024-05-01T12:53:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.968+0000 7f411552b800 20 remove_watcher() i=2 2024-05-01T12:53:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.969+0000 7f411552b800 20 remove_watcher() i=3 2024-05-01T12:53:42.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.970+0000 7f411552b800 20 remove_watcher() i=4 2024-05-01T12:53:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.971+0000 7f411552b800 20 remove_watcher() i=5 2024-05-01T12:53:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.973+0000 7f411552b800 20 remove_watcher() i=6 2024-05-01T12:53:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:42.974+0000 7f411552b800 20 remove_watcher() i=7 2024-05-01T12:53:43.023 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.023 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.060+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.061+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.061+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.062+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.063+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.063+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.064+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.064+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.064+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.064+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 zone MyZone found 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.065+0000 7f359e725800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.066+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.066+0000 7f359e725800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.066+0000 7f359e725800 20 started sync module instance, tier type = 2024-05-01T12:53:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.066+0000 7f359e725800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=0 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=1 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=2 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=3 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=4 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=5 2024-05-01T12:53:43.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.076+0000 7f359e725800 20 add_watcher() i=6 2024-05-01T12:53:43.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.077+0000 7f359e725800 20 add_watcher() i=7 2024-05-01T12:53:43.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.077+0000 7f359e725800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.078+0000 7f3450fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 5 note: GC not initialized 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f33cb7fe640 20 reqs_thread_entry: start 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 20 Filter name: none 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f33c97fa640 20 reqs_thread_entry: start 2024-05-01T12:53:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.079+0000 7f359e725800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : miss 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom info.flags=0x0 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 adding MyZone.rgw.log++pubsub.kaboom to cache LRU end 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom : hit (negative entry) 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_2 : miss 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 20 rados_obj.operate() r=0 bl.length=234 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_2 info.flags=0x15 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 adding MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_2 to cache LRU end 2024-05-01T12:53:43.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.080+0000 7f359e725800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_2 2024-05-01T12:53:43.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.081+0000 7f359e725800 10 removing MyZone.rgw.meta+topics+topic.kaboom:ashubx-2_topic_2 from cache 2024-05-01T12:53:43.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.081+0000 7f359e725800 10 distributing notification oid=MyZone.rgw.control:notify.6 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.kaboom:ashubx-2_topic_2, ofs0, ns] 2024-05-01T12:53:43.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.082+0000 7f347dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855372 cookie 94648515977696 notifier 4766 bl.length()=174 2024-05-01T12:53:43.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.083+0000 7f359e725800 20 WARNING: failed to remove topic buckets obj buckets.kaboom:ashubx-2_topic_2 with: (2) No such file or directory 2024-05-01T12:53:43.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.084+0000 7f3450fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.086+0000 7f359e725800 20 remove_watcher() i=0 2024-05-01T12:53:43.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.086+0000 7f359e725800 2 removed watcher, disabling cache 2024-05-01T12:53:43.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.087+0000 7f359e725800 20 remove_watcher() i=1 2024-05-01T12:53:43.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.088+0000 7f359e725800 20 remove_watcher() i=2 2024-05-01T12:53:43.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.089+0000 7f359e725800 20 remove_watcher() i=3 2024-05-01T12:53:43.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.090+0000 7f359e725800 20 remove_watcher() i=4 2024-05-01T12:53:43.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.091+0000 7f359e725800 20 remove_watcher() i=5 2024-05-01T12:53:43.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.092+0000 7f359e725800 20 remove_watcher() i=6 2024-05-01T12:53:43.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.093+0000 7f359e725800 20 remove_watcher() i=7 2024-05-01T12:53:43.141 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.141 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.177+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.178+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.178+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.179+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.179+0000 7fb96d30c800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.179+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.179+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.179+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.180+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.180+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.181+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.182+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.182+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 zone MyZone found 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 started sync module instance, tier type = 2024-05-01T12:53:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.183+0000 7fb96d30c800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=0 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=1 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=2 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=3 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=4 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=5 2024-05-01T12:53:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.194+0000 7fb96d30c800 20 add_watcher() i=6 2024-05-01T12:53:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.195+0000 7fb96d30c800 20 add_watcher() i=7 2024-05-01T12:53:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.195+0000 7fb96d30c800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb8217e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb96d30c800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb96d30c800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb96d30c800 5 note: GC not initialized 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb79bfff640 20 reqs_thread_entry: start 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb96d30c800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.197+0000 7fb96d30c800 20 Filter name: none 2024-05-01T12:53:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.198+0000 7fb799ffb640 20 reqs_thread_entry: start 2024-05-01T12:53:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.207+0000 7fb8217e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.209+0000 7fb96d30c800 20 remove_watcher() i=0 2024-05-01T12:53:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.209+0000 7fb96d30c800 2 removed watcher, disabling cache 2024-05-01T12:53:43.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.210+0000 7fb96d30c800 20 remove_watcher() i=1 2024-05-01T12:53:43.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.211+0000 7fb96d30c800 20 remove_watcher() i=2 2024-05-01T12:53:43.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.212+0000 7fb96d30c800 20 remove_watcher() i=3 2024-05-01T12:53:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.213+0000 7fb96d30c800 20 remove_watcher() i=4 2024-05-01T12:53:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.215+0000 7fb96d30c800 20 remove_watcher() i=5 2024-05-01T12:53:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.216+0000 7fb96d30c800 20 remove_watcher() i=6 2024-05-01T12:53:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.217+0000 7fb96d30c800 20 remove_watcher() i=7 2024-05-01T12:53:43.229 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:53:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 notification list/get/delete on master ... ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.317+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.318+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.318+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.319+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.319+0000 7f7fc392a800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.319+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.320+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.320+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.320+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.320+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.321+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.321+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.321+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.321+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.322+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.322+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.322+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.322+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 zone MyZone found 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.323+0000 7f7fc392a800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.324+0000 7f7fc392a800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.324+0000 7f7fc392a800 20 started sync module instance, tier type = 2024-05-01T12:53:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.324+0000 7f7fc392a800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=0 2024-05-01T12:53:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=1 2024-05-01T12:53:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=2 2024-05-01T12:53:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=3 2024-05-01T12:53:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=4 2024-05-01T12:53:43.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=5 2024-05-01T12:53:43.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.341+0000 7f7fc392a800 20 add_watcher() i=6 2024-05-01T12:53:43.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.342+0000 7f7fc392a800 20 add_watcher() i=7 2024-05-01T12:53:43.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.342+0000 7f7fc392a800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.343+0000 7f7e75fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7fc392a800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7fc392a800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7fc392a800 5 note: GC not initialized 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7e74fe1640 20 reqs_thread_entry: start 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7fc392a800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7fc392a800 20 Filter name: none 2024-05-01T12:53:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.344+0000 7f7dee7fc640 20 reqs_thread_entry: start 2024-05-01T12:53:43.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.354+0000 7f7e75fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.358+0000 7f7fc392a800 20 remove_watcher() i=0 2024-05-01T12:53:43.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.358+0000 7f7fc392a800 2 removed watcher, disabling cache 2024-05-01T12:53:43.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.360+0000 7f7fc392a800 20 remove_watcher() i=1 2024-05-01T12:53:43.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.361+0000 7f7fc392a800 20 remove_watcher() i=2 2024-05-01T12:53:43.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.362+0000 7f7fc392a800 20 remove_watcher() i=3 2024-05-01T12:53:43.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.363+0000 7f7fc392a800 20 remove_watcher() i=4 2024-05-01T12:53:43.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.364+0000 7f7fc392a800 20 remove_watcher() i=5 2024-05-01T12:53:43.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.365+0000 7f7fc392a800 20 remove_watcher() i=6 2024-05-01T12:53:43.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.366+0000 7f7fc392a800 20 remove_watcher() i=7 2024-05-01T12:53:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.488 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.524+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.525+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.525+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.526+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.526+0000 7f78cd410800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.526+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.526+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.526+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.527+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.527+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.527+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.528+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.528+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.528+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.528+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.528+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 zone MyZone found 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.529+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.530+0000 7f78cd410800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.530+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.530+0000 7f78cd410800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.530+0000 7f78cd410800 20 started sync module instance, tier type = 2024-05-01T12:53:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.530+0000 7f78cd410800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=0 2024-05-01T12:53:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=1 2024-05-01T12:53:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=2 2024-05-01T12:53:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=3 2024-05-01T12:53:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=4 2024-05-01T12:53:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=5 2024-05-01T12:53:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.544+0000 7f78cd410800 20 add_watcher() i=6 2024-05-01T12:53:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.546+0000 7f78cd410800 20 add_watcher() i=7 2024-05-01T12:53:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.546+0000 7f78cd410800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f777efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f78cd410800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f78cd410800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f78cd410800 5 note: GC not initialized 2024-05-01T12:53:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f777dfe3640 20 reqs_thread_entry: start 2024-05-01T12:53:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f78cd410800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.549+0000 7f78cd410800 20 Filter name: none 2024-05-01T12:53:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.550+0000 7f76ef7fe640 20 reqs_thread_entry: start 2024-05-01T12:53:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.550+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.550+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.550+0000 7f78cd410800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.551+0000 7f78cd410800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.551+0000 7f78cd410800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T12:53:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.552+0000 7f777efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.554+0000 7f78cd410800 20 remove_watcher() i=0 2024-05-01T12:53:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.554+0000 7f78cd410800 2 removed watcher, disabling cache 2024-05-01T12:53:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.555+0000 7f78cd410800 20 remove_watcher() i=1 2024-05-01T12:53:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.556+0000 7f78cd410800 20 remove_watcher() i=2 2024-05-01T12:53:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.557+0000 7f78cd410800 20 remove_watcher() i=3 2024-05-01T12:53:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.558+0000 7f78cd410800 20 remove_watcher() i=4 2024-05-01T12:53:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.559+0000 7f78cd410800 20 remove_watcher() i=5 2024-05-01T12:53:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.560+0000 7f78cd410800 20 remove_watcher() i=6 2024-05-01T12:53:43.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.563+0000 7f78cd410800 20 remove_watcher() i=7 2024-05-01T12:53:43.612 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.612 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.649+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.651+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.651+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.651+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.651+0000 7f3210f25800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.651+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.652+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.652+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.652+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.652+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.653+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.653+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.654+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 zone MyZone found 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.655+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.656+0000 7f3210f25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.656+0000 7f3210f25800 20 started sync module instance, tier type = 2024-05-01T12:53:43.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.656+0000 7f3210f25800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=0 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=1 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=2 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=3 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=4 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=5 2024-05-01T12:53:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.671+0000 7f3210f25800 20 add_watcher() i=6 2024-05-01T12:53:43.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.674+0000 7f3210f25800 20 add_watcher() i=7 2024-05-01T12:53:43.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.674+0000 7f3210f25800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f30c2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f3210f25800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f3210f25800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f3210f25800 5 note: GC not initialized 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f30c1fe3640 20 reqs_thread_entry: start 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.676+0000 7f3210f25800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.677+0000 7f3210f25800 20 Filter name: none 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.677+0000 7f30337fe640 20 reqs_thread_entry: start 2024-05-01T12:53:43.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.677+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.677+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:43.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.677+0000 7f3210f25800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.678+0000 7f3210f25800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:43.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.678+0000 7f3210f25800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T12:53:43.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.679+0000 7f30c2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.681+0000 7f3210f25800 20 remove_watcher() i=0 2024-05-01T12:53:43.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.681+0000 7f3210f25800 2 removed watcher, disabling cache 2024-05-01T12:53:43.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.682+0000 7f3210f25800 20 remove_watcher() i=1 2024-05-01T12:53:43.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.683+0000 7f3210f25800 20 remove_watcher() i=2 2024-05-01T12:53:43.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.684+0000 7f3210f25800 20 remove_watcher() i=3 2024-05-01T12:53:43.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.685+0000 7f3210f25800 20 remove_watcher() i=4 2024-05-01T12:53:43.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.686+0000 7f3210f25800 20 remove_watcher() i=5 2024-05-01T12:53:43.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.687+0000 7f3210f25800 20 remove_watcher() i=6 2024-05-01T12:53:43.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.690+0000 7f3210f25800 20 remove_watcher() i=7 2024-05-01T12:53:43.740 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.740 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.777+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.778+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.778+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.779+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.779+0000 7fa936e12800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.779+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.779+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.779+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.780+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.780+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.781+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 zone MyZone found 2024-05-01T12:53:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.782+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.783+0000 7fa936e12800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.783+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.783+0000 7fa936e12800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.783+0000 7fa936e12800 20 started sync module instance, tier type = 2024-05-01T12:53:43.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.783+0000 7fa936e12800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=0 2024-05-01T12:53:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=1 2024-05-01T12:53:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=2 2024-05-01T12:53:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=3 2024-05-01T12:53:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=4 2024-05-01T12:53:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=5 2024-05-01T12:53:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.800+0000 7fa936e12800 20 add_watcher() i=6 2024-05-01T12:53:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.802+0000 7fa936e12800 20 add_watcher() i=7 2024-05-01T12:53:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.802+0000 7fa936e12800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.804+0000 7fa7e8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 5 note: GC not initialized 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa7637fe640 20 reqs_thread_entry: start 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 20 Filter name: none 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa7617fa640 20 reqs_thread_entry: start 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 10 cache get: name=MyZone.rgw.meta+root+ashubx-3 : miss 2024-05-01T12:53:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.805+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:43.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 cache put: name=MyZone.rgw.meta+root+ashubx-3 info.flags=0x11 2024-05-01T12:53:43.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 adding MyZone.rgw.meta+root+ashubx-3 to cache LRU end 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 cache get: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 : miss 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 adding MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 updating xattr: name=user.rgw.bucket-notification bl.length()=1217 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.806+0000 7fa936e12800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 RGWRados::convert_old_bucket_info(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]) 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 cache get: name=MyZone.rgw.meta+root+ashubx-3 : type miss (requested=0x17, cached=0x11) 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 cache put: name=MyZone.rgw.meta+root+ashubx-3 info.flags=0x17 2024-05-01T12:53:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 moving MyZone.rgw.meta+root+ashubx-3 to cache LRU end 2024-05-01T12:53:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.807+0000 7fa936e12800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.808+0000 7fa936e12800 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.808+0000 7fa936e12800 10 moving MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.808+0000 7fa936e12800 10 updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.808+0000 7fa936e12800 10 updating xattr: name=user.rgw.bucket-notification bl.length()=800 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.808+0000 7fa936e12800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:root:.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1, ofs0, ns] 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa815ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855365 cookie 94502912936800 notifier 4847 bl.length()=1673 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa815ffb640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa815ffb640 10 rgw watcher librados: moving MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa815ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa815ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.bucket-notification bl.length()=800 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa936e12800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): orig_sources= new_sources= 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa936e12800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): potential sources added=[] removed=[] 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa936e12800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): orig_dests= new_dests= 2024-05-01T12:53:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.809+0000 7fa936e12800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): potential dests added=[] removed=[] 2024-05-01T12:53:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.810+0000 7fa7e8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.812+0000 7fa936e12800 20 remove_watcher() i=0 2024-05-01T12:53:43.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.812+0000 7fa936e12800 2 removed watcher, disabling cache 2024-05-01T12:53:43.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.813+0000 7fa936e12800 20 remove_watcher() i=1 2024-05-01T12:53:43.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.814+0000 7fa936e12800 20 remove_watcher() i=2 2024-05-01T12:53:43.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.815+0000 7fa936e12800 20 remove_watcher() i=3 2024-05-01T12:53:43.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.816+0000 7fa936e12800 20 remove_watcher() i=4 2024-05-01T12:53:43.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.817+0000 7fa936e12800 20 remove_watcher() i=5 2024-05-01T12:53:43.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.818+0000 7fa936e12800 20 remove_watcher() i=6 2024-05-01T12:53:43.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.820+0000 7fa936e12800 20 remove_watcher() i=7 2024-05-01T12:53:43.868 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.868 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:43.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.904+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.906+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.906+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.907+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.907+0000 7f6c7df46800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.907+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.907+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.907+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.908+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:43.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.908+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.908+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:43.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.909+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.909+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.909+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.909+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:43.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.909+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:43.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 zone MyZone found 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.910+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.911+0000 7f6c7df46800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.911+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.911+0000 7f6c7df46800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.911+0000 7f6c7df46800 20 started sync module instance, tier type = 2024-05-01T12:53:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.911+0000 7f6c7df46800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=0 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=1 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=2 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=3 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=4 2024-05-01T12:53:43.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.922+0000 7f6c7df46800 20 add_watcher() i=5 2024-05-01T12:53:43.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.923+0000 7f6c7df46800 20 add_watcher() i=6 2024-05-01T12:53:43.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.923+0000 7f6c7df46800 20 add_watcher() i=7 2024-05-01T12:53:43.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.923+0000 7f6c7df46800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:43.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.924+0000 7f6b31fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 5 note: GC not initialized 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6b30fe1640 20 reqs_thread_entry: start 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 20 Filter name: none 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6aaa7fc640 20 reqs_thread_entry: start 2024-05-01T12:53:43.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.925+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.926+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:43.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.926+0000 7f6c7df46800 20 rados->read ofs=0 len=0 2024-05-01T12:53:43.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.926+0000 7f6c7df46800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:43.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.926+0000 7f6c7df46800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T12:53:43.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.927+0000 7f6b31fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:43.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.929+0000 7f6c7df46800 20 remove_watcher() i=0 2024-05-01T12:53:43.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.929+0000 7f6c7df46800 2 removed watcher, disabling cache 2024-05-01T12:53:43.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.930+0000 7f6c7df46800 20 remove_watcher() i=1 2024-05-01T12:53:43.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.931+0000 7f6c7df46800 20 remove_watcher() i=2 2024-05-01T12:53:43.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.932+0000 7f6c7df46800 20 remove_watcher() i=3 2024-05-01T12:53:43.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.933+0000 7f6c7df46800 20 remove_watcher() i=4 2024-05-01T12:53:43.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.934+0000 7f6c7df46800 20 remove_watcher() i=5 2024-05-01T12:53:43.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.935+0000 7f6c7df46800 20 remove_watcher() i=6 2024-05-01T12:53:43.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:43.936+0000 7f6c7df46800 20 remove_watcher() i=7 2024-05-01T12:53:43.985 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:43.985 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:44.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.022+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.023+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.023+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.024+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.024+0000 7f6280f19800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:44.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.024+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.024+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.024+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.025+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:44.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.025+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:44.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.026+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.027+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.027+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:44.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 zone MyZone found 2024-05-01T12:53:44.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:44.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 started sync module instance, tier type = 2024-05-01T12:53:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.028+0000 7f6280f19800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=0 2024-05-01T12:53:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=1 2024-05-01T12:53:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=2 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=3 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=4 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.039+0000 7f6280f19800 20 add_watcher() i=5 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.040+0000 7f6280f19800 20 add_watcher() i=6 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.040+0000 7f6280f19800 20 add_watcher() i=7 2024-05-01T12:53:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.040+0000 7f6280f19800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f61357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f6280f19800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f6280f19800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f6280f19800 5 note: GC not initialized 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f60affff640 20 reqs_thread_entry: start 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f6280f19800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.042+0000 7f6280f19800 20 Filter name: none 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f60adffb640 20 reqs_thread_entry: start 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 cache get: name=MyZone.rgw.meta+root+ashubx-3 : miss 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 cache put: name=MyZone.rgw.meta+root+ashubx-3 info.flags=0x11 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 adding MyZone.rgw.meta+root+ashubx-3 to cache LRU end 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 cache get: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 : miss 2024-05-01T12:53:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:44.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:44.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 adding MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:44.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T12:53:44.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 updating xattr: name=user.rgw.bucket-notification bl.length()=800 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.043+0000 7f6280f19800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 RGWRados::convert_old_bucket_info(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]) 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 cache get: name=MyZone.rgw.meta+root+ashubx-3 : type miss (requested=0x17, cached=0x11) 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 cache put: name=MyZone.rgw.meta+root+ashubx-3 info.flags=0x17 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 moving MyZone.rgw.meta+root+ashubx-3 to cache LRU end 2024-05-01T12:53:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.044+0000 7f6280f19800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T12:53:44.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.045+0000 7f6280f19800 10 cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:44.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.045+0000 7f6280f19800 10 moving MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:44.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.045+0000 7f6280f19800 10 updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:44.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.045+0000 7f6280f19800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:root:.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1, ofs0, ns] 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f61627fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855366 cookie 94080619208880 notifier 4883 bl.length()=837 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f61627fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 info.flags=0x17 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f61627fc640 10 rgw watcher librados: moving MyZone.rgw.meta+root+.bucket.meta.ashubx-3:b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1 to cache LRU end 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f61627fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f6280f19800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): orig_sources= new_sources= 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f6280f19800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): potential sources added=[] removed=[] 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f6280f19800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): orig_dests= new_dests= 2024-05-01T12:53:44.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.046+0000 7f6280f19800 20 handle_bi_update(): bucket=:ashubx-3[b3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.1]): potential dests added=[] removed=[] 2024-05-01T12:53:44.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.047+0000 7f6280f19800 20 Successfully removed topic bucket mapping for bucket: ashubx-3 and topic: ashubx-3_topic_1 2024-05-01T12:53:44.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.048+0000 7f61357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:44.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.050+0000 7f6280f19800 20 remove_watcher() i=0 2024-05-01T12:53:44.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.050+0000 7f6280f19800 2 removed watcher, disabling cache 2024-05-01T12:53:44.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.051+0000 7f6280f19800 20 remove_watcher() i=1 2024-05-01T12:53:44.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.052+0000 7f6280f19800 20 remove_watcher() i=2 2024-05-01T12:53:44.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.053+0000 7f6280f19800 20 remove_watcher() i=3 2024-05-01T12:53:44.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.054+0000 7f6280f19800 20 remove_watcher() i=4 2024-05-01T12:53:44.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.055+0000 7f6280f19800 20 remove_watcher() i=5 2024-05-01T12:53:44.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.056+0000 7f6280f19800 20 remove_watcher() i=6 2024-05-01T12:53:44.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.057+0000 7f6280f19800 20 remove_watcher() i=7 2024-05-01T12:53:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:53:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:53:44.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.141+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.143+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.143+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:53:44.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.144+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:53:44.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.145+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.145+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:53:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.145+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.146+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.146+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.146+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:53:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.146+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:53:44.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:53:44.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 zone MyZone found 2024-05-01T12:53:44.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 period zonegroup name MyZoneGroup 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.147+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.148+0000 7fcddbe62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.148+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.148+0000 7fcddbe62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.148+0000 7fcddbe62800 20 started sync module instance, tier type = 2024-05-01T12:53:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.148+0000 7fcddbe62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=0 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=1 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=2 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=3 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=4 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=5 2024-05-01T12:53:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.163+0000 7fcddbe62800 20 add_watcher() i=6 2024-05-01T12:53:44.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.166+0000 7fcddbe62800 20 add_watcher() i=7 2024-05-01T12:53:44.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.166+0000 7fcddbe62800 2 all 8 watchers are set, enabling cache 2024-05-01T12:53:44.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.167+0000 7fcc837fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 5 note: GC not initialized 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcc827fc640 20 reqs_thread_entry: start 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 20 init_complete bucket index max shards: 11 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 20 Filter name: none 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcbfffff640 20 reqs_thread_entry: start 2024-05-01T12:53:44.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.168+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.169+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=186 2024-05-01T12:53:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.169+0000 7fcddbe62800 20 rados->read ofs=0 len=0 2024-05-01T12:53:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.169+0000 7fcddbe62800 20 rados_obj.operate() r=0 bl.length=400 2024-05-01T12:53:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.169+0000 7fcddbe62800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T12:53:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.170+0000 7fcc837fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:53:44.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.172+0000 7fcddbe62800 20 remove_watcher() i=0 2024-05-01T12:53:44.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.172+0000 7fcddbe62800 2 removed watcher, disabling cache 2024-05-01T12:53:44.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.173+0000 7fcddbe62800 20 remove_watcher() i=1 2024-05-01T12:53:44.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.174+0000 7fcddbe62800 20 remove_watcher() i=2 2024-05-01T12:53:44.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.175+0000 7fcddbe62800 20 remove_watcher() i=3 2024-05-01T12:53:44.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.176+0000 7fcddbe62800 20 remove_watcher() i=4 2024-05-01T12:53:44.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.177+0000 7fcddbe62800 20 remove_watcher() i=5 2024-05-01T12:53:44.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.178+0000 7fcddbe62800 20 remove_watcher() i=6 2024-05-01T12:53:44.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:53:44.179+0000 7fcddbe62800 20 remove_watcher() i=7 2024-05-01T12:53:44.191 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:53:44.361 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 notification set/get/delete on master ... ok 2024-05-01T12:53:44.491 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 notification set/get/delete on master with empty config ... ok 2024-05-01T12:53:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 notification set/get/delete on master ... ok 2024-05-01T12:53:44.783 INFO:teuthology.orchestra.run.smithi050.stderr:test deletion of multiple keys on master ... 127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.783 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.785 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.785 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.786 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.786 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.787 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:44.787 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:44] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:45.792 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:45] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:45.792 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:53:45] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:53:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:test pushing http s3 notification ... 172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.479 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.482 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.503 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.519 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:51.575 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:51] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.597 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.607 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.608 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.609 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.610 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:53:56.612 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.50 - - [01/May/2024 12:53:56] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:02.207 INFO:teuthology.orchestra.run.smithi050.stderr:test pushing cloudevents notification on master ... 127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.213 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.214 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.220 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:02.225 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:02] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.282 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:07] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:12.846 INFO:teuthology.orchestra.run.smithi050.stderr:test that opaque id set in topic, is sent in notification on master ... 127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.852 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.853 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.860 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:12.861 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:12] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:17] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:18.584 INFO:teuthology.orchestra.run.smithi050.stderr:test that when object is deleted due to lifecycle policy, notification is sent on master ... ignoring --setuser ceph since I am not root 2024-05-01T12:54:18.584 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.623+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.624+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.624+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.625+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.625+0000 7f1e3a928800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.625+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.626+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.626+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.626+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.626+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.627+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.627+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.627+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.627+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.628+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.628+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.628+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.628+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 20 zone MyZone found 2024-05-01T12:54:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.629+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.630+0000 7f1e3a928800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.630+0000 7f1e3a928800 20 rados->read ofs=0 len=0 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.630+0000 7f1e3a928800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.630+0000 7f1e3a928800 20 started sync module instance, tier type = 2024-05-01T12:54:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.630+0000 7f1e3a928800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:18.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=0 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=1 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=2 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=3 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=4 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=5 2024-05-01T12:54:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.644+0000 7f1e3a928800 20 add_watcher() i=6 2024-05-01T12:54:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.645+0000 7f1e3a928800 20 add_watcher() i=7 2024-05-01T12:54:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.645+0000 7f1e3a928800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:18.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.647+0000 7f1cecfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:18.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.648+0000 7f1e3a928800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:18.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.648+0000 7f1e3a928800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.684+0000 7f1c677fe640 20 reqs_thread_entry: start 2024-05-01T12:54:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.684+0000 7f1e3a928800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.684+0000 7f1e3a928800 20 Filter name: none 2024-05-01T12:54:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.685+0000 7f1c657fa640 20 reqs_thread_entry: start 2024-05-01T12:54:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=0, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=1, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=0, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=1, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.725+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.726+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=2, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.726+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=2, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.726+0000 7f1d19ffb640 20 handle_completion(): completion ok for obj=ooo5 2024-05-01T12:54:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.726+0000 7f1d197fa640 20 handle_completion(): completion ok for obj=ooo1 2024-05-01T12:54:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1d19ffb640 20 handle_completion(): completion ok for obj=ooo9 2024-05-01T12:54:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=3, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=3, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=4, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.727+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=4, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.728+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.728+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=5, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.728+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=5, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.731+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.731+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=6, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.731+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=6, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=7, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=7, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=8, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=8, curl_handle=0x7f1c440047f0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=9, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=9, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.732+0000 7f1d197fa640 20 handle_completion(): completion ok for obj=ooo2 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.733+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.733+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=10, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.733+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=10, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.733+0000 7f1d19ffb640 20 handle_completion(): completion ok for obj=ooo6 2024-05-01T12:54:18.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.733+0000 7f1d197fa640 20 handle_completion(): completion ok for obj=ooo7 2024-05-01T12:54:18.735 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.734+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.734+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=11, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.734+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=11, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.736+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.736+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=12, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.736+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=12, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c52ffd640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c52ffd640 20 register_request mgr=0x55be25f253a0 req_data->id=13, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c657fa640 20 link_request req_data=0x7f1c3c00b7e0 req_data->id=13, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=14, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=14, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.737+0000 7f1d19ffb640 20 handle_completion(): completion ok for obj=ooo4 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=15, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=15, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=16, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=16, curl_handle=0x7f1c3c005de0 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c537fe640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c537fe640 20 register_request mgr=0x55be25f253a0 req_data->id=17, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1c657fa640 20 link_request req_data=0x7f1c44008900 req_data->id=17, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.738+0000 7f1d197fa640 20 handle_completion(): completion ok for obj=ooo3 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.739+0000 7f1d19ffb640 20 handle_completion(): completion ok for obj=ooo0 2024-05-01T12:54:18.740 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.741+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.741+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=18, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.741+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=18, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.742+0000 7f1c53fff640 20 sending request to http://localhost:13053 2024-05-01T12:54:18.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.742+0000 7f1c53fff640 20 register_request mgr=0x55be25f253a0 req_data->id=19, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.742+0000 7f1c657fa640 20 link_request req_data=0x7f1c4c00a2e0 req_data->id=19, curl_handle=0x7f1c4c0031a0 2024-05-01T12:54:18.743 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:18] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:18.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:18.743+0000 7f1d197fa640 20 handle_completion(): completion ok for obj=ooo8 2024-05-01T12:54:20.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.745+0000 7f1cecfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:20.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.748+0000 7f1e3a928800 20 remove_watcher() i=0 2024-05-01T12:54:20.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.748+0000 7f1e3a928800 2 removed watcher, disabling cache 2024-05-01T12:54:20.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.749+0000 7f1e3a928800 20 remove_watcher() i=1 2024-05-01T12:54:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.750+0000 7f1e3a928800 20 remove_watcher() i=2 2024-05-01T12:54:20.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.751+0000 7f1e3a928800 20 remove_watcher() i=3 2024-05-01T12:54:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.753+0000 7f1e3a928800 20 remove_watcher() i=4 2024-05-01T12:54:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.754+0000 7f1e3a928800 20 remove_watcher() i=5 2024-05-01T12:54:20.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.755+0000 7f1e3a928800 20 remove_watcher() i=6 2024-05-01T12:54:20.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:20.757+0000 7f1e3a928800 20 remove_watcher() i=7 2024-05-01T12:54:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:27.589 INFO:teuthology.orchestra.run.smithi050.stderr:test that when a multipart upload is aborted by lifecycle policy, notification is sent on master ... ignoring --setuser ceph since I am not root 2024-05-01T12:54:27.590 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.630+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.631+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.631+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.632+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.632+0000 7f8b56416800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.632+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.633+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.633+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.633+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.633+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.634+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.634+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.635+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.635+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.635+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.635+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 zone MyZone found 2024-05-01T12:54:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.636+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.637+0000 7f8b56416800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.637+0000 7f8b56416800 20 rados->read ofs=0 len=0 2024-05-01T12:54:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.637+0000 7f8b56416800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.637+0000 7f8b56416800 20 started sync module instance, tier type = 2024-05-01T12:54:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.637+0000 7f8b56416800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=0 2024-05-01T12:54:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=1 2024-05-01T12:54:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=2 2024-05-01T12:54:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=3 2024-05-01T12:54:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=4 2024-05-01T12:54:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=5 2024-05-01T12:54:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.649+0000 7f8b56416800 20 add_watcher() i=6 2024-05-01T12:54:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.651+0000 7f8b56416800 20 add_watcher() i=7 2024-05-01T12:54:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.651+0000 7f8b56416800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.652+0000 7f8a037fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.653+0000 7f8b56416800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.653+0000 7f8b56416800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.690+0000 7f8a027fc640 20 reqs_thread_entry: start 2024-05-01T12:54:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.690+0000 7f8b56416800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.690+0000 7f8b56416800 20 Filter name: none 2024-05-01T12:54:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.690+0000 7f8973fff640 20 reqs_thread_entry: start 2024-05-01T12:54:27.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:27.704+0000 7f8b56416800 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.23 head last stored at Wed May 1 12:54:18 2024 2024-05-01T12:54:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.125+0000 7f8a037fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.128+0000 7f8b56416800 20 remove_watcher() i=0 2024-05-01T12:54:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.128+0000 7f8b56416800 2 removed watcher, disabling cache 2024-05-01T12:54:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.129+0000 7f8b56416800 20 remove_watcher() i=1 2024-05-01T12:54:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.131+0000 7f8b56416800 20 remove_watcher() i=2 2024-05-01T12:54:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.132+0000 7f8b56416800 20 remove_watcher() i=3 2024-05-01T12:54:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.133+0000 7f8b56416800 20 remove_watcher() i=4 2024-05-01T12:54:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.134+0000 7f8b56416800 20 remove_watcher() i=5 2024-05-01T12:54:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.136+0000 7f8b56416800 20 remove_watcher() i=6 2024-05-01T12:54:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:29.137+0000 7f8b56416800 20 remove_watcher() i=7 2024-05-01T12:54:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:test http multipart object upload on master ... 127.0.0.1 - - [01/May/2024 12:54:49] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:54.859 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 12:54:54] "POST / HTTP/1.1" 200 - 2024-05-01T12:54:55.361 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:test persistent topic stats ... ignoring --setuser ceph since I am not root 2024-05-01T12:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:55.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.490+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.492+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:55.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.492+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.493+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.493+0000 7fcf72746800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.493+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.493+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.493+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.494+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.494+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.494+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.494+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.495+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.495+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.495+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:55.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.495+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:55.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 zone MyZone found 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:55.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.496+0000 7fcf72746800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 started sync module instance, tier type = 2024-05-01T12:54:55.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.497+0000 7fcf72746800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=0 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=1 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=2 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=3 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=4 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=5 2024-05-01T12:54:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.509+0000 7fcf72746800 20 add_watcher() i=6 2024-05-01T12:54:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.510+0000 7fcf72746800 20 add_watcher() i=7 2024-05-01T12:54:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.510+0000 7fcf72746800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fce26fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcf72746800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcf72746800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcf72746800 5 note: GC not initialized 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fce25fdb640 20 reqs_thread_entry: start 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcf72746800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcf72746800 20 Filter name: none 2024-05-01T12:54:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.512+0000 7fcd977fe640 20 reqs_thread_entry: start 2024-05-01T12:54:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.513+0000 7fcf72746800 20 rados->read ofs=0 len=0 2024-05-01T12:54:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.513+0000 7fcf72746800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:54:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.515+0000 7fce26fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.517+0000 7fcf72746800 20 remove_watcher() i=0 2024-05-01T12:54:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.517+0000 7fcf72746800 2 removed watcher, disabling cache 2024-05-01T12:54:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.518+0000 7fcf72746800 20 remove_watcher() i=1 2024-05-01T12:54:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.519+0000 7fcf72746800 20 remove_watcher() i=2 2024-05-01T12:54:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.520+0000 7fcf72746800 20 remove_watcher() i=3 2024-05-01T12:54:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.521+0000 7fcf72746800 20 remove_watcher() i=4 2024-05-01T12:54:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.522+0000 7fcf72746800 20 remove_watcher() i=5 2024-05-01T12:54:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.524+0000 7fcf72746800 20 remove_watcher() i=6 2024-05-01T12:54:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:55.525+0000 7fcf72746800 20 remove_watcher() i=7 2024-05-01T12:54:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:54:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.990+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.991+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.991+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.992+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.992+0000 7f12eb51f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.992+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.992+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.992+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.993+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.993+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.993+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.993+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.994+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.994+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.994+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.994+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 zone MyZone found 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.995+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.996+0000 7f12eb51f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.996+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.996+0000 7f12eb51f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.996+0000 7f12eb51f800 20 started sync module instance, tier type = 2024-05-01T12:54:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:57.996+0000 7f12eb51f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=0 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=1 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=2 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=3 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=4 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=5 2024-05-01T12:54:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.006+0000 7f12eb51f800 20 add_watcher() i=6 2024-05-01T12:54:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.007+0000 7f12eb51f800 20 add_watcher() i=7 2024-05-01T12:54:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.007+0000 7f12eb51f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f119dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f12eb51f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f12eb51f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f12eb51f800 5 note: GC not initialized 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f119cfe1640 20 reqs_thread_entry: start 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f12eb51f800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f12eb51f800 20 Filter name: none 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.009+0000 7f11167fc640 20 reqs_thread_entry: start 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.010+0000 7f12eb51f800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.010+0000 7f12eb51f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:54:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.011+0000 7f119dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.013+0000 7f12eb51f800 20 remove_watcher() i=0 2024-05-01T12:54:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.013+0000 7f12eb51f800 2 removed watcher, disabling cache 2024-05-01T12:54:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.014+0000 7f12eb51f800 20 remove_watcher() i=1 2024-05-01T12:54:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.015+0000 7f12eb51f800 20 remove_watcher() i=2 2024-05-01T12:54:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.016+0000 7f12eb51f800 20 remove_watcher() i=3 2024-05-01T12:54:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.017+0000 7f12eb51f800 20 remove_watcher() i=4 2024-05-01T12:54:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.018+0000 7f12eb51f800 20 remove_watcher() i=5 2024-05-01T12:54:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.019+0000 7f12eb51f800 20 remove_watcher() i=6 2024-05-01T12:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.020+0000 7f12eb51f800 20 remove_watcher() i=7 2024-05-01T12:54:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:54:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:58.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.312+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.314+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.314+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.315+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.315+0000 7f178ee10800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.315+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.315+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.315+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.316+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.316+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.316+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.316+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.317+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.317+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.317+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.317+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.318+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.318+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 zone MyZone found 2024-05-01T12:54:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.319+0000 7f178ee10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.320+0000 7f178ee10800 20 started sync module instance, tier type = 2024-05-01T12:54:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.320+0000 7f178ee10800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=0 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=1 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=2 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=3 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=4 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=5 2024-05-01T12:54:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.330+0000 7f178ee10800 20 add_watcher() i=6 2024-05-01T12:54:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.331+0000 7f178ee10800 20 add_watcher() i=7 2024-05-01T12:54:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.331+0000 7f178ee10800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.332+0000 7f1640fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 5 note: GC not initialized 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f15bb7fe640 20 reqs_thread_entry: start 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 20 Filter name: none 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f15b97fa640 20 reqs_thread_entry: start 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.333+0000 7f178ee10800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.334+0000 7f178ee10800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:54:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.335+0000 7f1640fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.338+0000 7f178ee10800 20 remove_watcher() i=0 2024-05-01T12:54:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.338+0000 7f178ee10800 2 removed watcher, disabling cache 2024-05-01T12:54:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.339+0000 7f178ee10800 20 remove_watcher() i=1 2024-05-01T12:54:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.340+0000 7f178ee10800 20 remove_watcher() i=2 2024-05-01T12:54:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.341+0000 7f178ee10800 20 remove_watcher() i=3 2024-05-01T12:54:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.342+0000 7f178ee10800 20 remove_watcher() i=4 2024-05-01T12:54:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.343+0000 7f178ee10800 20 remove_watcher() i=5 2024-05-01T12:54:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.345+0000 7f178ee10800 20 remove_watcher() i=6 2024-05-01T12:54:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.346+0000 7f178ee10800 20 remove_watcher() i=7 2024-05-01T12:54:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:54:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:54:58.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.437+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.439+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.439+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.440+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.440+0000 7f0f5c7a2800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.440+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.440+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.440+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.441+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:54:58.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.441+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:54:58.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:54:58.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.442+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.443+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:54:58.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.443+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 zone MyZone found 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 period zonegroup name MyZoneGroup 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:58.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:54:58.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 started sync module instance, tier type = 2024-05-01T12:54:58.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.444+0000 7f0f5c7a2800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=0 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=1 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=2 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=3 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=4 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=5 2024-05-01T12:54:58.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.456+0000 7f0f5c7a2800 20 add_watcher() i=6 2024-05-01T12:54:58.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.457+0000 7f0f5c7a2800 20 add_watcher() i=7 2024-05-01T12:54:58.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.457+0000 7f0f5c7a2800 2 all 8 watchers are set, enabling cache 2024-05-01T12:54:58.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.458+0000 7f0e10fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0f5c7a2800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0f5c7a2800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0f5c7a2800 5 note: GC not initialized 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0d8b7fe640 20 reqs_thread_entry: start 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0f5c7a2800 20 init_complete bucket index max shards: 11 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0f5c7a2800 20 Filter name: none 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.459+0000 7f0d897fa640 20 reqs_thread_entry: start 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.460+0000 7f0f5c7a2800 20 rados->read ofs=0 len=0 2024-05-01T12:54:58.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.460+0000 7f0f5c7a2800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:54:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.461+0000 7f0e10fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:54:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.463+0000 7f0f5c7a2800 20 remove_watcher() i=0 2024-05-01T12:54:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.463+0000 7f0f5c7a2800 2 removed watcher, disabling cache 2024-05-01T12:54:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.464+0000 7f0f5c7a2800 20 remove_watcher() i=1 2024-05-01T12:54:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.465+0000 7f0f5c7a2800 20 remove_watcher() i=2 2024-05-01T12:54:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.466+0000 7f0f5c7a2800 20 remove_watcher() i=3 2024-05-01T12:54:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.467+0000 7f0f5c7a2800 20 remove_watcher() i=4 2024-05-01T12:54:58.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.469+0000 7f0f5c7a2800 20 remove_watcher() i=5 2024-05-01T12:54:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.469+0000 7f0f5c7a2800 20 remove_watcher() i=6 2024-05-01T12:54:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:54:58.470+0000 7f0f5c7a2800 20 remove_watcher() i=7 2024-05-01T12:55:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.579+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.581+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.581+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.582+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.582+0000 7fa38f3a2800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.582+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.582+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.582+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.583+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.583+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.584+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.584+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.584+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.584+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.585+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.585+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.585+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.585+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 20 zone MyZone found 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.586+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.587+0000 7fa38f3a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.587+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.587+0000 7fa38f3a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.587+0000 7fa38f3a2800 20 started sync module instance, tier type = 2024-05-01T12:55:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.587+0000 7fa38f3a2800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=0 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=1 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=2 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=3 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=4 2024-05-01T12:55:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=5 2024-05-01T12:55:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.599+0000 7fa38f3a2800 20 add_watcher() i=6 2024-05-01T12:55:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.600+0000 7fa38f3a2800 20 add_watcher() i=7 2024-05-01T12:55:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.600+0000 7fa38f3a2800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.602+0000 7fa242fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 5 note: GC not initialized 2024-05-01T12:55:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa241fe3640 20 reqs_thread_entry: start 2024-05-01T12:55:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 20 Filter name: none 2024-05-01T12:55:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa1b37fe640 20 reqs_thread_entry: start 2024-05-01T12:55:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.603+0000 7fa38f3a2800 20 rados->read ofs=0 len=0 2024-05-01T12:55:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.604+0000 7fa38f3a2800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.605+0000 7fa242fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.608+0000 7fa38f3a2800 20 remove_watcher() i=0 2024-05-01T12:55:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.608+0000 7fa38f3a2800 2 removed watcher, disabling cache 2024-05-01T12:55:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.609+0000 7fa38f3a2800 20 remove_watcher() i=1 2024-05-01T12:55:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.610+0000 7fa38f3a2800 20 remove_watcher() i=2 2024-05-01T12:55:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.612+0000 7fa38f3a2800 20 remove_watcher() i=3 2024-05-01T12:55:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.613+0000 7fa38f3a2800 20 remove_watcher() i=4 2024-05-01T12:55:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.614+0000 7fa38f3a2800 20 remove_watcher() i=5 2024-05-01T12:55:03.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.615+0000 7fa38f3a2800 20 remove_watcher() i=6 2024-05-01T12:55:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:03.617+0000 7fa38f3a2800 20 remove_watcher() i=7 2024-05-01T12:55:08.680 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:08.681 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.719+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.720+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.720+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.721+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.721+0000 7f9b06733800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.721+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.722+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.722+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.723+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:08.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.723+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.723+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.723+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.724+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.724+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.724+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.724+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:08.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 zone MyZone found 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.725+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.726+0000 7f9b06733800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.726+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.726+0000 7f9b06733800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.726+0000 7f9b06733800 20 started sync module instance, tier type = 2024-05-01T12:55:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.726+0000 7f9b06733800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=0 2024-05-01T12:55:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=1 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=2 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=3 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=4 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=5 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.737+0000 7f9b06733800 20 add_watcher() i=6 2024-05-01T12:55:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.738+0000 7f9b06733800 20 add_watcher() i=7 2024-05-01T12:55:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.738+0000 7f9b06733800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:08.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f99b8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f9b06733800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f9b06733800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f9b06733800 5 note: GC not initialized 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f99337fe640 20 reqs_thread_entry: start 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f9b06733800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f9b06733800 20 Filter name: none 2024-05-01T12:55:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.741+0000 7f99317fa640 20 reqs_thread_entry: start 2024-05-01T12:55:08.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.742+0000 7f9b06733800 20 rados->read ofs=0 len=0 2024-05-01T12:55:08.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.742+0000 7f9b06733800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.744+0000 7f99b8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.746+0000 7f9b06733800 20 remove_watcher() i=0 2024-05-01T12:55:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.746+0000 7f9b06733800 2 removed watcher, disabling cache 2024-05-01T12:55:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.747+0000 7f9b06733800 20 remove_watcher() i=1 2024-05-01T12:55:08.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.748+0000 7f9b06733800 20 remove_watcher() i=2 2024-05-01T12:55:08.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.749+0000 7f9b06733800 20 remove_watcher() i=3 2024-05-01T12:55:08.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.750+0000 7f9b06733800 20 remove_watcher() i=4 2024-05-01T12:55:08.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.751+0000 7f9b06733800 20 remove_watcher() i=5 2024-05-01T12:55:08.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.753+0000 7f9b06733800 20 remove_watcher() i=6 2024-05-01T12:55:08.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:08.754+0000 7f9b06733800 20 remove_watcher() i=7 2024-05-01T12:55:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.848+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.849+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.850+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.850+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.850+0000 7f9e6bb1f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.851+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.851+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.851+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.852+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.852+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.852+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.852+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.853+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.853+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.853+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.853+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 zone MyZone found 2024-05-01T12:55:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.854+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.855+0000 7f9e6bb1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.855+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.855+0000 7f9e6bb1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.855+0000 7f9e6bb1f800 20 started sync module instance, tier type = 2024-05-01T12:55:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.855+0000 7f9e6bb1f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=0 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=1 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=2 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=3 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=4 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=5 2024-05-01T12:55:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.867+0000 7f9e6bb1f800 20 add_watcher() i=6 2024-05-01T12:55:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.869+0000 7f9e6bb1f800 20 add_watcher() i=7 2024-05-01T12:55:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.869+0000 7f9e6bb1f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9d1dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9e6bb1f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9e6bb1f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9e6bb1f800 5 note: GC not initialized 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9d1cfe1640 20 reqs_thread_entry: start 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.871+0000 7f9e6bb1f800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.872+0000 7f9e6bb1f800 20 Filter name: none 2024-05-01T12:55:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.872+0000 7f9c967fc640 20 reqs_thread_entry: start 2024-05-01T12:55:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.872+0000 7f9e6bb1f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.872+0000 7f9e6bb1f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.874+0000 7f9d1dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.876+0000 7f9e6bb1f800 20 remove_watcher() i=0 2024-05-01T12:55:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.876+0000 7f9e6bb1f800 2 removed watcher, disabling cache 2024-05-01T12:55:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.877+0000 7f9e6bb1f800 20 remove_watcher() i=1 2024-05-01T12:55:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.878+0000 7f9e6bb1f800 20 remove_watcher() i=2 2024-05-01T12:55:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.879+0000 7f9e6bb1f800 20 remove_watcher() i=3 2024-05-01T12:55:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.880+0000 7f9e6bb1f800 20 remove_watcher() i=4 2024-05-01T12:55:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.881+0000 7f9e6bb1f800 20 remove_watcher() i=5 2024-05-01T12:55:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.882+0000 7f9e6bb1f800 20 remove_watcher() i=6 2024-05-01T12:55:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:13.883+0000 7f9e6bb1f800 20 remove_watcher() i=7 2024-05-01T12:55:18.944 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:18.944 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:18.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.981+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.982+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.982+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.983+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.983+0000 7f645d298800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.983+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.983+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.983+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.984+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.984+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.985+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.985+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.985+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.985+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.986+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.986+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.986+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.986+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 zone MyZone found 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.987+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.988+0000 7f645d298800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.988+0000 7f645d298800 20 started sync module instance, tier type = 2024-05-01T12:55:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.988+0000 7f645d298800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=0 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=1 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=2 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=3 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=4 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=5 2024-05-01T12:55:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:18.999+0000 7f645d298800 20 add_watcher() i=6 2024-05-01T12:55:19.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.000+0000 7f645d298800 20 add_watcher() i=7 2024-05-01T12:55:19.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.000+0000 7f645d298800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:19.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f63117e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:19.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f645d298800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:19.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f645d298800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f645d298800 5 note: GC not initialized 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f628bfff640 20 reqs_thread_entry: start 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.002+0000 7f645d298800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.003+0000 7f645d298800 20 Filter name: none 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.003+0000 7f6289ffb640 20 reqs_thread_entry: start 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.003+0000 7f645d298800 20 rados->read ofs=0 len=0 2024-05-01T12:55:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.003+0000 7f645d298800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:19.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.005+0000 7f63117e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:19.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.007+0000 7f645d298800 20 remove_watcher() i=0 2024-05-01T12:55:19.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.007+0000 7f645d298800 2 removed watcher, disabling cache 2024-05-01T12:55:19.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.008+0000 7f645d298800 20 remove_watcher() i=1 2024-05-01T12:55:19.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.009+0000 7f645d298800 20 remove_watcher() i=2 2024-05-01T12:55:19.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.010+0000 7f645d298800 20 remove_watcher() i=3 2024-05-01T12:55:19.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.011+0000 7f645d298800 20 remove_watcher() i=4 2024-05-01T12:55:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.012+0000 7f645d298800 20 remove_watcher() i=5 2024-05-01T12:55:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.013+0000 7f645d298800 20 remove_watcher() i=6 2024-05-01T12:55:19.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:19.014+0000 7f645d298800 20 remove_watcher() i=7 2024-05-01T12:55:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:24.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.113+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.115+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.115+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.116+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.116+0000 7f9819d19800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.116+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.116+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.116+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.117+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.117+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.117+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.117+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.118+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.118+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.118+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:24.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.118+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:24.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:24.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 zone MyZone found 2024-05-01T12:55:24.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.119+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.120+0000 7f9819d19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.120+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.120+0000 7f9819d19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.120+0000 7f9819d19800 20 started sync module instance, tier type = 2024-05-01T12:55:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.120+0000 7f9819d19800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=0 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=1 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=2 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=3 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=4 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.130+0000 7f9819d19800 20 add_watcher() i=5 2024-05-01T12:55:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.131+0000 7f9819d19800 20 add_watcher() i=6 2024-05-01T12:55:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.132+0000 7f9819d19800 20 add_watcher() i=7 2024-05-01T12:55:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.132+0000 7f9819d19800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f96cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f9819d19800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f9819d19800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f9819d19800 5 note: GC not initialized 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f96ccfe1640 20 reqs_thread_entry: start 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.134+0000 7f9819d19800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.135+0000 7f9819d19800 20 Filter name: none 2024-05-01T12:55:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.135+0000 7f96467fc640 20 reqs_thread_entry: start 2024-05-01T12:55:24.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.135+0000 7f9819d19800 20 rados->read ofs=0 len=0 2024-05-01T12:55:24.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.135+0000 7f9819d19800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:24.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.137+0000 7f96cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:24.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.139+0000 7f9819d19800 20 remove_watcher() i=0 2024-05-01T12:55:24.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.139+0000 7f9819d19800 2 removed watcher, disabling cache 2024-05-01T12:55:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.140+0000 7f9819d19800 20 remove_watcher() i=1 2024-05-01T12:55:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.141+0000 7f9819d19800 20 remove_watcher() i=2 2024-05-01T12:55:24.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.142+0000 7f9819d19800 20 remove_watcher() i=3 2024-05-01T12:55:24.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.143+0000 7f9819d19800 20 remove_watcher() i=4 2024-05-01T12:55:24.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.144+0000 7f9819d19800 20 remove_watcher() i=5 2024-05-01T12:55:24.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.145+0000 7f9819d19800 20 remove_watcher() i=6 2024-05-01T12:55:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:24.146+0000 7f9819d19800 20 remove_watcher() i=7 2024-05-01T12:55:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:29.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.244+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.245+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:29.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.245+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.246+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:29.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.246+0000 7f4251687800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:29.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.246+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.246+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:29.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.246+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.247+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:29.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.247+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.248+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.249+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.249+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 zone MyZone found 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.250+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.251+0000 7f4251687800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:29.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.251+0000 7f4251687800 20 started sync module instance, tier type = 2024-05-01T12:55:29.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.251+0000 7f4251687800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=0 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=1 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=2 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=3 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=4 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=5 2024-05-01T12:55:29.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.263+0000 7f4251687800 20 add_watcher() i=6 2024-05-01T12:55:29.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.264+0000 7f4251687800 20 add_watcher() i=7 2024-05-01T12:55:29.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.264+0000 7f4251687800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:29.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.265+0000 7f4105fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 5 note: GC not initialized 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4104fe1640 20 reqs_thread_entry: start 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 20 Filter name: none 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f407e7fc640 20 reqs_thread_entry: start 2024-05-01T12:55:29.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.266+0000 7f4251687800 20 rados->read ofs=0 len=0 2024-05-01T12:55:29.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.267+0000 7f4251687800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:29.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.268+0000 7f4105fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.271+0000 7f4251687800 20 remove_watcher() i=0 2024-05-01T12:55:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.271+0000 7f4251687800 2 removed watcher, disabling cache 2024-05-01T12:55:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.272+0000 7f4251687800 20 remove_watcher() i=1 2024-05-01T12:55:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.273+0000 7f4251687800 20 remove_watcher() i=2 2024-05-01T12:55:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.274+0000 7f4251687800 20 remove_watcher() i=3 2024-05-01T12:55:29.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.275+0000 7f4251687800 20 remove_watcher() i=4 2024-05-01T12:55:29.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.276+0000 7f4251687800 20 remove_watcher() i=5 2024-05-01T12:55:29.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.277+0000 7f4251687800 20 remove_watcher() i=6 2024-05-01T12:55:29.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:29.278+0000 7f4251687800 20 remove_watcher() i=7 2024-05-01T12:55:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.377+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.378+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.378+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.379+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.379+0000 7f6108f2b800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.379+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.380+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.380+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.380+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.380+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.381+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.381+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.381+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.381+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.382+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.382+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.382+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.382+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 zone MyZone found 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.383+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.384+0000 7f6108f2b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.384+0000 7f6108f2b800 20 started sync module instance, tier type = 2024-05-01T12:55:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.384+0000 7f6108f2b800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=0 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=1 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=2 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=3 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=4 2024-05-01T12:55:34.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.395+0000 7f6108f2b800 20 add_watcher() i=5 2024-05-01T12:55:34.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.396+0000 7f6108f2b800 20 add_watcher() i=6 2024-05-01T12:55:34.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.396+0000 7f6108f2b800 20 add_watcher() i=7 2024-05-01T12:55:34.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.396+0000 7f6108f2b800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:34.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.398+0000 7f5fbafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.398+0000 7f6108f2b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.398+0000 7f6108f2b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.398+0000 7f6108f2b800 5 note: GC not initialized 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.398+0000 7f5fb9fe3640 20 reqs_thread_entry: start 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.399+0000 7f6108f2b800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.399+0000 7f6108f2b800 20 Filter name: none 2024-05-01T12:55:34.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.399+0000 7f5f2b7fe640 20 reqs_thread_entry: start 2024-05-01T12:55:34.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.399+0000 7f6108f2b800 20 rados->read ofs=0 len=0 2024-05-01T12:55:34.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.399+0000 7f6108f2b800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:34.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.401+0000 7f5fbafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:34.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.403+0000 7f6108f2b800 20 remove_watcher() i=0 2024-05-01T12:55:34.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.403+0000 7f6108f2b800 2 removed watcher, disabling cache 2024-05-01T12:55:34.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.404+0000 7f6108f2b800 20 remove_watcher() i=1 2024-05-01T12:55:34.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.405+0000 7f6108f2b800 20 remove_watcher() i=2 2024-05-01T12:55:34.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.406+0000 7f6108f2b800 20 remove_watcher() i=3 2024-05-01T12:55:34.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.407+0000 7f6108f2b800 20 remove_watcher() i=4 2024-05-01T12:55:34.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.409+0000 7f6108f2b800 20 remove_watcher() i=5 2024-05-01T12:55:34.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.410+0000 7f6108f2b800 20 remove_watcher() i=6 2024-05-01T12:55:34.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:34.411+0000 7f6108f2b800 20 remove_watcher() i=7 2024-05-01T12:55:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:39.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.510+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.512+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:39.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.512+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.512+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.512+0000 7f627bb2f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.512+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.513+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.513+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.513+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.513+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.514+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.514+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.514+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.514+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.515+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.515+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.515+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.515+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 zone MyZone found 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.516+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.517+0000 7f627bb2f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.517+0000 7f627bb2f800 20 started sync module instance, tier type = 2024-05-01T12:55:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.517+0000 7f627bb2f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=0 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=1 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=2 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=3 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=4 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=5 2024-05-01T12:55:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.528+0000 7f627bb2f800 20 add_watcher() i=6 2024-05-01T12:55:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.529+0000 7f627bb2f800 20 add_watcher() i=7 2024-05-01T12:55:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.529+0000 7f627bb2f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f612dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f627bb2f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f627bb2f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f627bb2f800 5 note: GC not initialized 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f612cfe1640 20 reqs_thread_entry: start 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f627bb2f800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f627bb2f800 20 Filter name: none 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.531+0000 7f60a67fc640 20 reqs_thread_entry: start 2024-05-01T12:55:39.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.532+0000 7f627bb2f800 20 rados->read ofs=0 len=0 2024-05-01T12:55:39.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.532+0000 7f627bb2f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:39.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.533+0000 7f612dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:39.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.535+0000 7f627bb2f800 20 remove_watcher() i=0 2024-05-01T12:55:39.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.535+0000 7f627bb2f800 2 removed watcher, disabling cache 2024-05-01T12:55:39.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.536+0000 7f627bb2f800 20 remove_watcher() i=1 2024-05-01T12:55:39.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.537+0000 7f627bb2f800 20 remove_watcher() i=2 2024-05-01T12:55:39.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.538+0000 7f627bb2f800 20 remove_watcher() i=3 2024-05-01T12:55:39.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.539+0000 7f627bb2f800 20 remove_watcher() i=4 2024-05-01T12:55:39.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.540+0000 7f627bb2f800 20 remove_watcher() i=5 2024-05-01T12:55:39.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.541+0000 7f627bb2f800 20 remove_watcher() i=6 2024-05-01T12:55:39.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:39.543+0000 7f627bb2f800 20 remove_watcher() i=7 2024-05-01T12:55:44.598 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:44.598 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.634+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.636+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:44.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.636+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.637+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:44.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.637+0000 7f1487012800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.637+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.637+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.637+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.638+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.638+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:44.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.639+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.640+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.640+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 zone MyZone found 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 started sync module instance, tier type = 2024-05-01T12:55:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.641+0000 7f1487012800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=0 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=1 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=2 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=3 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=4 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=5 2024-05-01T12:55:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.652+0000 7f1487012800 20 add_watcher() i=6 2024-05-01T12:55:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.654+0000 7f1487012800 20 add_watcher() i=7 2024-05-01T12:55:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.654+0000 7f1487012800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.655+0000 7f1338fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 5 note: GC not initialized 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f12b37fe640 20 reqs_thread_entry: start 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 20 Filter name: none 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f12b17fa640 20 reqs_thread_entry: start 2024-05-01T12:55:44.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.656+0000 7f1487012800 20 rados->read ofs=0 len=0 2024-05-01T12:55:44.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.657+0000 7f1487012800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:44.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.658+0000 7f1338fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.660+0000 7f1487012800 20 remove_watcher() i=0 2024-05-01T12:55:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.660+0000 7f1487012800 2 removed watcher, disabling cache 2024-05-01T12:55:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.661+0000 7f1487012800 20 remove_watcher() i=1 2024-05-01T12:55:44.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.662+0000 7f1487012800 20 remove_watcher() i=2 2024-05-01T12:55:44.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.663+0000 7f1487012800 20 remove_watcher() i=3 2024-05-01T12:55:44.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.665+0000 7f1487012800 20 remove_watcher() i=4 2024-05-01T12:55:44.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.666+0000 7f1487012800 20 remove_watcher() i=5 2024-05-01T12:55:44.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.667+0000 7f1487012800 20 remove_watcher() i=6 2024-05-01T12:55:44.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:44.668+0000 7f1487012800 20 remove_watcher() i=7 2024-05-01T12:55:49.725 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:49.725 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.763+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.764+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.764+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.764+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.764+0000 7f62de735800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.764+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.765+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.765+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:49.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.766+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.767+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.767+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.767+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:49.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.767+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 zone MyZone found 2024-05-01T12:55:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.768+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.769+0000 7f62de735800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.769+0000 7f62de735800 20 started sync module instance, tier type = 2024-05-01T12:55:49.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.769+0000 7f62de735800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:49.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=0 2024-05-01T12:55:49.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=1 2024-05-01T12:55:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=2 2024-05-01T12:55:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=3 2024-05-01T12:55:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=4 2024-05-01T12:55:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.780+0000 7f62de735800 20 add_watcher() i=5 2024-05-01T12:55:49.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.782+0000 7f62de735800 20 add_watcher() i=6 2024-05-01T12:55:49.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.782+0000 7f62de735800 20 add_watcher() i=7 2024-05-01T12:55:49.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.782+0000 7f62de735800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:49.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.784+0000 7f6190fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 5 note: GC not initialized 2024-05-01T12:55:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f610b7fe640 20 reqs_thread_entry: start 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 20 Filter name: none 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f61097fa640 20 reqs_thread_entry: start 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.785+0000 7f62de735800 20 rados->read ofs=0 len=0 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.786+0000 7f62de735800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.787+0000 7f6190fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.789+0000 7f62de735800 20 remove_watcher() i=0 2024-05-01T12:55:49.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.789+0000 7f62de735800 2 removed watcher, disabling cache 2024-05-01T12:55:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.790+0000 7f62de735800 20 remove_watcher() i=1 2024-05-01T12:55:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.791+0000 7f62de735800 20 remove_watcher() i=2 2024-05-01T12:55:49.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.792+0000 7f62de735800 20 remove_watcher() i=3 2024-05-01T12:55:49.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.793+0000 7f62de735800 20 remove_watcher() i=4 2024-05-01T12:55:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.794+0000 7f62de735800 20 remove_watcher() i=5 2024-05-01T12:55:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.796+0000 7f62de735800 20 remove_watcher() i=6 2024-05-01T12:55:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:49.797+0000 7f62de735800 20 remove_watcher() i=7 2024-05-01T12:55:54.858 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:54.859 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:55:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.894+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.896+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.896+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.897+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.897+0000 7f124fd27800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:55:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.897+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.897+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:54.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.897+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.898+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:55:54.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.898+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.898+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.898+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.899+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.899+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.899+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:55:54.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.899+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:55:54.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 zone MyZone found 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 period zonegroup name MyZoneGroup 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:55:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.900+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.901+0000 7f124fd27800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:54.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.901+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.901+0000 7f124fd27800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:55:54.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.901+0000 7f124fd27800 20 started sync module instance, tier type = 2024-05-01T12:55:54.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.901+0000 7f124fd27800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=0 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=1 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=2 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=3 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=4 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=5 2024-05-01T12:55:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.912+0000 7f124fd27800 20 add_watcher() i=6 2024-05-01T12:55:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.913+0000 7f124fd27800 20 add_watcher() i=7 2024-05-01T12:55:54.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.913+0000 7f124fd27800 2 all 8 watchers are set, enabling cache 2024-05-01T12:55:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.914+0000 7f1101fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 5 note: GC not initialized 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f1100fe1640 20 reqs_thread_entry: start 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 20 init_complete bucket index max shards: 11 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 20 Filter name: none 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f107a7fc640 20 reqs_thread_entry: start 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.915+0000 7f124fd27800 20 rados->read ofs=0 len=0 2024-05-01T12:55:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.916+0000 7f124fd27800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:55:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.917+0000 7f1101fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:55:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.919+0000 7f124fd27800 20 remove_watcher() i=0 2024-05-01T12:55:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.919+0000 7f124fd27800 2 removed watcher, disabling cache 2024-05-01T12:55:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.920+0000 7f124fd27800 20 remove_watcher() i=1 2024-05-01T12:55:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.921+0000 7f124fd27800 20 remove_watcher() i=2 2024-05-01T12:55:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.922+0000 7f124fd27800 20 remove_watcher() i=3 2024-05-01T12:55:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.923+0000 7f124fd27800 20 remove_watcher() i=4 2024-05-01T12:55:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.925+0000 7f124fd27800 20 remove_watcher() i=5 2024-05-01T12:55:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.926+0000 7f124fd27800 20 remove_watcher() i=6 2024-05-01T12:55:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:55:54.927+0000 7f124fd27800 20 remove_watcher() i=7 2024-05-01T12:55:59.988 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:55:59.988 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.024+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.026+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.026+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.026+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.026+0000 7f885bfa2800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.026+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.027+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.027+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.027+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:00.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.027+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.028+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:00.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.028+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.029+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:00.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.029+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.029+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:00.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.029+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:00.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 zone MyZone found 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.030+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.031+0000 7f885bfa2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.031+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.031+0000 7f885bfa2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.031+0000 7f885bfa2800 20 started sync module instance, tier type = 2024-05-01T12:56:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.031+0000 7f885bfa2800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=0 2024-05-01T12:56:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=1 2024-05-01T12:56:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=2 2024-05-01T12:56:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=3 2024-05-01T12:56:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=4 2024-05-01T12:56:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=5 2024-05-01T12:56:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.042+0000 7f885bfa2800 20 add_watcher() i=6 2024-05-01T12:56:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.043+0000 7f885bfa2800 20 add_watcher() i=7 2024-05-01T12:56:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.043+0000 7f885bfa2800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.044+0000 7f870b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f885bfa2800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f885bfa2800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f885bfa2800 5 note: GC not initialized 2024-05-01T12:56:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f870a7fc640 20 reqs_thread_entry: start 2024-05-01T12:56:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f885bfa2800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f885bfa2800 20 Filter name: none 2024-05-01T12:56:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.045+0000 7f867bfff640 20 reqs_thread_entry: start 2024-05-01T12:56:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.046+0000 7f885bfa2800 20 rados->read ofs=0 len=0 2024-05-01T12:56:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.046+0000 7f885bfa2800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:00.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.047+0000 7f870b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.049+0000 7f885bfa2800 20 remove_watcher() i=0 2024-05-01T12:56:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.049+0000 7f885bfa2800 2 removed watcher, disabling cache 2024-05-01T12:56:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.050+0000 7f885bfa2800 20 remove_watcher() i=1 2024-05-01T12:56:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.051+0000 7f885bfa2800 20 remove_watcher() i=2 2024-05-01T12:56:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.052+0000 7f885bfa2800 20 remove_watcher() i=3 2024-05-01T12:56:00.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.053+0000 7f885bfa2800 20 remove_watcher() i=4 2024-05-01T12:56:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.054+0000 7f885bfa2800 20 remove_watcher() i=5 2024-05-01T12:56:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.055+0000 7f885bfa2800 20 remove_watcher() i=6 2024-05-01T12:56:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:00.056+0000 7f885bfa2800 20 remove_watcher() i=7 2024-05-01T12:56:05.117 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:05.118 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.154+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.156+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.156+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.157+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.157+0000 7f671a33e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.157+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.157+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.157+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.158+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.158+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.159+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.160+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.160+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 zone MyZone found 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 started sync module instance, tier type = 2024-05-01T12:56:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.161+0000 7f671a33e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=0 2024-05-01T12:56:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=1 2024-05-01T12:56:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=2 2024-05-01T12:56:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=3 2024-05-01T12:56:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=4 2024-05-01T12:56:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=5 2024-05-01T12:56:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.173+0000 7f671a33e800 20 add_watcher() i=6 2024-05-01T12:56:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.175+0000 7f671a33e800 20 add_watcher() i=7 2024-05-01T12:56:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.175+0000 7f671a33e800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f65cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f671a33e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f671a33e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f671a33e800 5 note: GC not initialized 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f65ccfe1640 20 reqs_thread_entry: start 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f671a33e800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.177+0000 7f671a33e800 20 Filter name: none 2024-05-01T12:56:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.178+0000 7f65467fc640 20 reqs_thread_entry: start 2024-05-01T12:56:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.178+0000 7f671a33e800 20 rados->read ofs=0 len=0 2024-05-01T12:56:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.178+0000 7f671a33e800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.180+0000 7f65cdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.182+0000 7f671a33e800 20 remove_watcher() i=0 2024-05-01T12:56:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.182+0000 7f671a33e800 2 removed watcher, disabling cache 2024-05-01T12:56:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.183+0000 7f671a33e800 20 remove_watcher() i=1 2024-05-01T12:56:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.184+0000 7f671a33e800 20 remove_watcher() i=2 2024-05-01T12:56:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.185+0000 7f671a33e800 20 remove_watcher() i=3 2024-05-01T12:56:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.186+0000 7f671a33e800 20 remove_watcher() i=4 2024-05-01T12:56:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.187+0000 7f671a33e800 20 remove_watcher() i=5 2024-05-01T12:56:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.188+0000 7f671a33e800 20 remove_watcher() i=6 2024-05-01T12:56:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:05.189+0000 7f671a33e800 20 remove_watcher() i=7 2024-05-01T12:56:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:10.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.285+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.287+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.287+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.288+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.288+0000 7f2655333800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.288+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.288+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.288+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.289+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.289+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.289+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.289+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.290+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.290+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.290+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.290+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 zone MyZone found 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.291+0000 7f2655333800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 started sync module instance, tier type = 2024-05-01T12:56:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.292+0000 7f2655333800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=0 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=1 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=2 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=3 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=4 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=5 2024-05-01T12:56:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.302+0000 7f2655333800 20 add_watcher() i=6 2024-05-01T12:56:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.303+0000 7f2655333800 20 add_watcher() i=7 2024-05-01T12:56:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.304+0000 7f2655333800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2506fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2655333800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2655333800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2655333800 5 note: GC not initialized 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2505fe3640 20 reqs_thread_entry: start 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2655333800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f2655333800 20 Filter name: none 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.306+0000 7f24777fe640 20 reqs_thread_entry: start 2024-05-01T12:56:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.307+0000 7f2655333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.307+0000 7f2655333800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.308+0000 7f2506fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.311+0000 7f2655333800 20 remove_watcher() i=0 2024-05-01T12:56:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.311+0000 7f2655333800 2 removed watcher, disabling cache 2024-05-01T12:56:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.312+0000 7f2655333800 20 remove_watcher() i=1 2024-05-01T12:56:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.313+0000 7f2655333800 20 remove_watcher() i=2 2024-05-01T12:56:10.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.314+0000 7f2655333800 20 remove_watcher() i=3 2024-05-01T12:56:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.315+0000 7f2655333800 20 remove_watcher() i=4 2024-05-01T12:56:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.316+0000 7f2655333800 20 remove_watcher() i=5 2024-05-01T12:56:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.317+0000 7f2655333800 20 remove_watcher() i=6 2024-05-01T12:56:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:10.318+0000 7f2655333800 20 remove_watcher() i=7 2024-05-01T12:56:15.379 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:15.379 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:15.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.416+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.418+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:15.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.418+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.419+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.419+0000 7fa7aa230800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.419+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.419+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.419+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.420+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.420+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.420+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.420+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.421+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.421+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.421+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:15.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.421+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:15.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 zone MyZone found 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.422+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.423+0000 7fa7aa230800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:15.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.423+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.423+0000 7fa7aa230800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:15.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.423+0000 7fa7aa230800 20 started sync module instance, tier type = 2024-05-01T12:56:15.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.423+0000 7fa7aa230800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=0 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=1 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=2 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=3 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=4 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=5 2024-05-01T12:56:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.433+0000 7fa7aa230800 20 add_watcher() i=6 2024-05-01T12:56:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.435+0000 7fa7aa230800 20 add_watcher() i=7 2024-05-01T12:56:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.435+0000 7fa7aa230800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:15.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.436+0000 7fa65b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa7aa230800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa7aa230800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa7aa230800 5 note: GC not initialized 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa65a7e4640 20 reqs_thread_entry: start 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa7aa230800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa7aa230800 20 Filter name: none 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.437+0000 7fa5cbfff640 20 reqs_thread_entry: start 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.438+0000 7fa7aa230800 20 rados->read ofs=0 len=0 2024-05-01T12:56:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.438+0000 7fa7aa230800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.439+0000 7fa65b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.441+0000 7fa7aa230800 20 remove_watcher() i=0 2024-05-01T12:56:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.441+0000 7fa7aa230800 2 removed watcher, disabling cache 2024-05-01T12:56:15.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.442+0000 7fa7aa230800 20 remove_watcher() i=1 2024-05-01T12:56:15.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.443+0000 7fa7aa230800 20 remove_watcher() i=2 2024-05-01T12:56:15.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.444+0000 7fa7aa230800 20 remove_watcher() i=3 2024-05-01T12:56:15.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.445+0000 7fa7aa230800 20 remove_watcher() i=4 2024-05-01T12:56:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.446+0000 7fa7aa230800 20 remove_watcher() i=5 2024-05-01T12:56:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.447+0000 7fa7aa230800 20 remove_watcher() i=6 2024-05-01T12:56:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:15.448+0000 7fa7aa230800 20 remove_watcher() i=7 2024-05-01T12:56:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:20.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.544+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.546+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:20.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.546+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.547+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.548+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.548+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.548+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.548+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.549+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.549+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.549+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.549+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 20 zone MyZone found 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.550+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.551+0000 7fe034012800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.551+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.551+0000 7fe034012800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.551+0000 7fe034012800 20 started sync module instance, tier type = 2024-05-01T12:56:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.551+0000 7fe034012800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=0 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=1 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=2 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=3 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=4 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=5 2024-05-01T12:56:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.561+0000 7fe034012800 20 add_watcher() i=6 2024-05-01T12:56:20.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.563+0000 7fe034012800 20 add_watcher() i=7 2024-05-01T12:56:20.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.563+0000 7fe034012800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:20.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.564+0000 7fdee5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fe034012800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fe034012800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fe034012800 5 note: GC not initialized 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fdee4fe1640 20 reqs_thread_entry: start 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fe034012800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fe034012800 20 Filter name: none 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.565+0000 7fde5e7fc640 20 reqs_thread_entry: start 2024-05-01T12:56:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.566+0000 7fe034012800 20 rados->read ofs=0 len=0 2024-05-01T12:56:20.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.566+0000 7fe034012800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:20.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.568+0000 7fdee5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:20.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.570+0000 7fe034012800 20 remove_watcher() i=0 2024-05-01T12:56:20.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.570+0000 7fe034012800 2 removed watcher, disabling cache 2024-05-01T12:56:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.571+0000 7fe034012800 20 remove_watcher() i=1 2024-05-01T12:56:20.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.572+0000 7fe034012800 20 remove_watcher() i=2 2024-05-01T12:56:20.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.573+0000 7fe034012800 20 remove_watcher() i=3 2024-05-01T12:56:20.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.574+0000 7fe034012800 20 remove_watcher() i=4 2024-05-01T12:56:20.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.575+0000 7fe034012800 20 remove_watcher() i=5 2024-05-01T12:56:20.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.576+0000 7fe034012800 20 remove_watcher() i=6 2024-05-01T12:56:20.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:20.577+0000 7fe034012800 20 remove_watcher() i=7 2024-05-01T12:56:25.639 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:25.639 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:25.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.676+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.678+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:25.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.678+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.679+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:25.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.679+0000 7fa54bc30800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:25.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.679+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.679+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:25.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.679+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.680+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:25.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.680+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.680+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.680+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:25.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:25.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.681+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 zone MyZone found 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.682+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.683+0000 7fa54bc30800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.683+0000 7fa54bc30800 20 started sync module instance, tier type = 2024-05-01T12:56:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.683+0000 7fa54bc30800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=0 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=1 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=2 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=3 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=4 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=5 2024-05-01T12:56:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.693+0000 7fa54bc30800 20 add_watcher() i=6 2024-05-01T12:56:25.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.694+0000 7fa54bc30800 20 add_watcher() i=7 2024-05-01T12:56:25.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.694+0000 7fa54bc30800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:25.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.696+0000 7fa3fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:25.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:25.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 5 note: GC not initialized 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa377fff640 20 reqs_thread_entry: start 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 20 Filter name: none 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa375ffb640 20 reqs_thread_entry: start 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.697+0000 7fa54bc30800 20 rados->read ofs=0 len=0 2024-05-01T12:56:25.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.698+0000 7fa54bc30800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:25.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.699+0000 7fa3fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:25.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.701+0000 7fa54bc30800 20 remove_watcher() i=0 2024-05-01T12:56:25.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.701+0000 7fa54bc30800 2 removed watcher, disabling cache 2024-05-01T12:56:25.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.702+0000 7fa54bc30800 20 remove_watcher() i=1 2024-05-01T12:56:25.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.703+0000 7fa54bc30800 20 remove_watcher() i=2 2024-05-01T12:56:25.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.704+0000 7fa54bc30800 20 remove_watcher() i=3 2024-05-01T12:56:25.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.705+0000 7fa54bc30800 20 remove_watcher() i=4 2024-05-01T12:56:25.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.706+0000 7fa54bc30800 20 remove_watcher() i=5 2024-05-01T12:56:25.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.708+0000 7fa54bc30800 20 remove_watcher() i=6 2024-05-01T12:56:25.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:25.709+0000 7fa54bc30800 20 remove_watcher() i=7 2024-05-01T12:56:30.768 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.805+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.807+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.807+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.807+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.807+0000 7fac06a0c800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.807+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.808+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.808+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.808+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.808+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.809+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.809+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.810+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 zone MyZone found 2024-05-01T12:56:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.811+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.812+0000 7fac06a0c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.812+0000 7fac06a0c800 20 started sync module instance, tier type = 2024-05-01T12:56:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.812+0000 7fac06a0c800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=0 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=1 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=2 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=3 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=4 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=5 2024-05-01T12:56:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.823+0000 7fac06a0c800 20 add_watcher() i=6 2024-05-01T12:56:30.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.824+0000 7fac06a0c800 20 add_watcher() i=7 2024-05-01T12:56:30.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.824+0000 7fac06a0c800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:30.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7faabafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7fac06a0c800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7fac06a0c800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7fac06a0c800 5 note: GC not initialized 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7faab9fdb640 20 reqs_thread_entry: start 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7fac06a0c800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7fac06a0c800 20 Filter name: none 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.826+0000 7faa2b7fe640 20 reqs_thread_entry: start 2024-05-01T12:56:30.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.827+0000 7fac06a0c800 20 rados->read ofs=0 len=0 2024-05-01T12:56:30.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.827+0000 7fac06a0c800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:30.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.828+0000 7faabafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:30.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.831+0000 7fac06a0c800 20 remove_watcher() i=0 2024-05-01T12:56:30.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.831+0000 7fac06a0c800 2 removed watcher, disabling cache 2024-05-01T12:56:30.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.832+0000 7fac06a0c800 20 remove_watcher() i=1 2024-05-01T12:56:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.833+0000 7fac06a0c800 20 remove_watcher() i=2 2024-05-01T12:56:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.834+0000 7fac06a0c800 20 remove_watcher() i=3 2024-05-01T12:56:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.835+0000 7fac06a0c800 20 remove_watcher() i=4 2024-05-01T12:56:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.836+0000 7fac06a0c800 20 remove_watcher() i=5 2024-05-01T12:56:30.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.837+0000 7fac06a0c800 20 remove_watcher() i=6 2024-05-01T12:56:30.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:30.838+0000 7fac06a0c800 20 remove_watcher() i=7 2024-05-01T12:56:35.898 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:35.898 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.935+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.936+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.936+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.937+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.937+0000 7f0a5c830800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.937+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.937+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.937+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.938+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.938+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.939+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.940+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.940+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 zone MyZone found 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:35.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 started sync module instance, tier type = 2024-05-01T12:56:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.941+0000 7f0a5c830800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=0 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=1 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=2 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=3 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=4 2024-05-01T12:56:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=5 2024-05-01T12:56:35.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.953+0000 7f0a5c830800 20 add_watcher() i=6 2024-05-01T12:56:35.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.954+0000 7f0a5c830800 20 add_watcher() i=7 2024-05-01T12:56:35.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.954+0000 7f0a5c830800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f090dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f0a5c830800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f0a5c830800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f0a5c830800 5 note: GC not initialized 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f090cfe1640 20 reqs_thread_entry: start 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f0a5c830800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.956+0000 7f0a5c830800 20 Filter name: none 2024-05-01T12:56:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.957+0000 7f08867fc640 20 reqs_thread_entry: start 2024-05-01T12:56:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.957+0000 7f0a5c830800 20 rados->read ofs=0 len=0 2024-05-01T12:56:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.957+0000 7f0a5c830800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:35.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.959+0000 7f090dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:35.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.961+0000 7f0a5c830800 20 remove_watcher() i=0 2024-05-01T12:56:35.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.961+0000 7f0a5c830800 2 removed watcher, disabling cache 2024-05-01T12:56:35.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.962+0000 7f0a5c830800 20 remove_watcher() i=1 2024-05-01T12:56:35.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.963+0000 7f0a5c830800 20 remove_watcher() i=2 2024-05-01T12:56:35.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.964+0000 7f0a5c830800 20 remove_watcher() i=3 2024-05-01T12:56:35.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.965+0000 7f0a5c830800 20 remove_watcher() i=4 2024-05-01T12:56:35.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.966+0000 7f0a5c830800 20 remove_watcher() i=5 2024-05-01T12:56:35.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.967+0000 7f0a5c830800 20 remove_watcher() i=6 2024-05-01T12:56:35.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:35.968+0000 7f0a5c830800 20 remove_watcher() i=7 2024-05-01T12:56:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.063+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.065+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.065+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.066+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.066+0000 7f4b5b498800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.066+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.066+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.066+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.067+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.067+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.068+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.068+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.068+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.068+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.069+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.069+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.069+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.069+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 zone MyZone found 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.070+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.071+0000 7f4b5b498800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.071+0000 7f4b5b498800 20 started sync module instance, tier type = 2024-05-01T12:56:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.071+0000 7f4b5b498800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=0 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=1 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=2 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=3 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=4 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=5 2024-05-01T12:56:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.084+0000 7f4b5b498800 20 add_watcher() i=6 2024-05-01T12:56:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.085+0000 7f4b5b498800 20 add_watcher() i=7 2024-05-01T12:56:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.085+0000 7f4b5b498800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.087+0000 7f4a0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4b5b498800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4b5b498800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4b5b498800 5 note: GC not initialized 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4a0dfe3640 20 reqs_thread_entry: start 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4b5b498800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f4b5b498800 20 Filter name: none 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.088+0000 7f497f7fe640 20 reqs_thread_entry: start 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.089+0000 7f4b5b498800 20 rados->read ofs=0 len=0 2024-05-01T12:56:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.089+0000 7f4b5b498800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.090+0000 7f4a0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.092+0000 7f4b5b498800 20 remove_watcher() i=0 2024-05-01T12:56:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.092+0000 7f4b5b498800 2 removed watcher, disabling cache 2024-05-01T12:56:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.093+0000 7f4b5b498800 20 remove_watcher() i=1 2024-05-01T12:56:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.094+0000 7f4b5b498800 20 remove_watcher() i=2 2024-05-01T12:56:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.095+0000 7f4b5b498800 20 remove_watcher() i=3 2024-05-01T12:56:41.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.096+0000 7f4b5b498800 20 remove_watcher() i=4 2024-05-01T12:56:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.097+0000 7f4b5b498800 20 remove_watcher() i=5 2024-05-01T12:56:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.098+0000 7f4b5b498800 20 remove_watcher() i=6 2024-05-01T12:56:41.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:41.099+0000 7f4b5b498800 20 remove_watcher() i=7 2024-05-01T12:56:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:46.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.199+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.201+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:46.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.201+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.202+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:46.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.202+0000 7f9495213800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:46.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.202+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.203+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:46.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.203+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.203+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:46.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.203+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.204+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:46.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.204+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.204+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.204+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.205+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.205+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.205+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.205+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 zone MyZone found 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:46.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.206+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.207+0000 7f9495213800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:46.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.207+0000 7f9495213800 20 started sync module instance, tier type = 2024-05-01T12:56:46.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.207+0000 7f9495213800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=0 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=1 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=2 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=3 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=4 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=5 2024-05-01T12:56:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.217+0000 7f9495213800 20 add_watcher() i=6 2024-05-01T12:56:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.218+0000 7f9495213800 20 add_watcher() i=7 2024-05-01T12:56:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.218+0000 7f9495213800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:46.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.220+0000 7f9346fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:46.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.220+0000 7f9495213800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:46.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.220+0000 7f9495213800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:46.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.220+0000 7f9495213800 5 note: GC not initialized 2024-05-01T12:56:46.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f9345fe3640 20 reqs_thread_entry: start 2024-05-01T12:56:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f9495213800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f9495213800 20 Filter name: none 2024-05-01T12:56:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f92b77fe640 20 reqs_thread_entry: start 2024-05-01T12:56:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f9495213800 20 rados->read ofs=0 len=0 2024-05-01T12:56:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.221+0000 7f9495213800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.223+0000 7f9346fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.225+0000 7f9495213800 20 remove_watcher() i=0 2024-05-01T12:56:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.225+0000 7f9495213800 2 removed watcher, disabling cache 2024-05-01T12:56:46.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.226+0000 7f9495213800 20 remove_watcher() i=1 2024-05-01T12:56:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.227+0000 7f9495213800 20 remove_watcher() i=2 2024-05-01T12:56:46.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.228+0000 7f9495213800 20 remove_watcher() i=3 2024-05-01T12:56:46.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.229+0000 7f9495213800 20 remove_watcher() i=4 2024-05-01T12:56:46.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.230+0000 7f9495213800 20 remove_watcher() i=5 2024-05-01T12:56:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.231+0000 7f9495213800 20 remove_watcher() i=6 2024-05-01T12:56:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:46.232+0000 7f9495213800 20 remove_watcher() i=7 2024-05-01T12:56:51.291 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:51.291 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:51.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.327+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.329+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:51.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.329+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.330+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:51.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.330+0000 7f67b053f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.330+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.330+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.331+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.331+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:51.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.331+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.332+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:51.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.332+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.332+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.332+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.333+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.333+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.333+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:51.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.333+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 zone MyZone found 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:51.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.334+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.335+0000 7f67b053f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.335+0000 7f67b053f800 20 started sync module instance, tier type = 2024-05-01T12:56:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.335+0000 7f67b053f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:51.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=0 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=1 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=2 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=3 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=4 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=5 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.346+0000 7f67b053f800 20 add_watcher() i=6 2024-05-01T12:56:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.347+0000 7f67b053f800 20 add_watcher() i=7 2024-05-01T12:56:51.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.347+0000 7f67b053f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:51.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.349+0000 7f6662fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.349+0000 7f67b053f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.349+0000 7f67b053f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.349+0000 7f67b053f800 5 note: GC not initialized 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.350+0000 7f6661fdb640 20 reqs_thread_entry: start 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.350+0000 7f67b053f800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.350+0000 7f67b053f800 20 Filter name: none 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.350+0000 7f65d37fe640 20 reqs_thread_entry: start 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.350+0000 7f67b053f800 20 rados->read ofs=0 len=0 2024-05-01T12:56:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.351+0000 7f67b053f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:51.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.352+0000 7f6662fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:51.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.354+0000 7f67b053f800 20 remove_watcher() i=0 2024-05-01T12:56:51.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.354+0000 7f67b053f800 2 removed watcher, disabling cache 2024-05-01T12:56:51.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.355+0000 7f67b053f800 20 remove_watcher() i=1 2024-05-01T12:56:51.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.356+0000 7f67b053f800 20 remove_watcher() i=2 2024-05-01T12:56:51.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.357+0000 7f67b053f800 20 remove_watcher() i=3 2024-05-01T12:56:51.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.358+0000 7f67b053f800 20 remove_watcher() i=4 2024-05-01T12:56:51.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.359+0000 7f67b053f800 20 remove_watcher() i=5 2024-05-01T12:56:51.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.361+0000 7f67b053f800 20 remove_watcher() i=6 2024-05-01T12:56:51.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:51.362+0000 7f67b053f800 20 remove_watcher() i=7 2024-05-01T12:56:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:56:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:56:56.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.458+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.460+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:56.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.460+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.461+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:56.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.461+0000 7fe2c9333800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:56:56.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.461+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.461+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:56.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.461+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.462+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:56:56.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.462+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:56:56.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:56.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:56:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.463+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.464+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:56:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.464+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:56:56.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 zone MyZone found 2024-05-01T12:56:56.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:56:56.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 period zonegroup name MyZoneGroup 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:56:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 started sync module instance, tier type = 2024-05-01T12:56:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.465+0000 7fe2c9333800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:56:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=0 2024-05-01T12:56:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=1 2024-05-01T12:56:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=2 2024-05-01T12:56:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=3 2024-05-01T12:56:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=4 2024-05-01T12:56:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=5 2024-05-01T12:56:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.476+0000 7fe2c9333800 20 add_watcher() i=6 2024-05-01T12:56:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.477+0000 7fe2c9333800 20 add_watcher() i=7 2024-05-01T12:56:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.477+0000 7fe2c9333800 2 all 8 watchers are set, enabling cache 2024-05-01T12:56:56.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe17afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe2c9333800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe2c9333800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe2c9333800 5 note: GC not initialized 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe179fe3640 20 reqs_thread_entry: start 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.479+0000 7fe2c9333800 20 init_complete bucket index max shards: 11 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.480+0000 7fe2c9333800 20 Filter name: none 2024-05-01T12:56:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.480+0000 7fe0eb7fe640 20 reqs_thread_entry: start 2024-05-01T12:56:56.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.480+0000 7fe2c9333800 20 rados->read ofs=0 len=0 2024-05-01T12:56:56.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.480+0000 7fe2c9333800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:56:56.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.482+0000 7fe17afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:56:56.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.484+0000 7fe2c9333800 20 remove_watcher() i=0 2024-05-01T12:56:56.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.484+0000 7fe2c9333800 2 removed watcher, disabling cache 2024-05-01T12:56:56.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.485+0000 7fe2c9333800 20 remove_watcher() i=1 2024-05-01T12:56:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.486+0000 7fe2c9333800 20 remove_watcher() i=2 2024-05-01T12:56:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.487+0000 7fe2c9333800 20 remove_watcher() i=3 2024-05-01T12:56:56.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.488+0000 7fe2c9333800 20 remove_watcher() i=4 2024-05-01T12:56:56.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.489+0000 7fe2c9333800 20 remove_watcher() i=5 2024-05-01T12:56:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.490+0000 7fe2c9333800 20 remove_watcher() i=6 2024-05-01T12:56:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:56:56.491+0000 7fe2c9333800 20 remove_watcher() i=7 2024-05-01T12:57:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:01.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.589+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.591+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:01.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.591+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.591+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:01.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.591+0000 7fdc35321800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:01.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.592+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.592+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.592+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:01.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.593+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.594+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:01.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.594+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:01.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 zone MyZone found 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.595+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.596+0000 7fdc35321800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:01.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.596+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.596+0000 7fdc35321800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:01.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.596+0000 7fdc35321800 20 started sync module instance, tier type = 2024-05-01T12:57:01.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.596+0000 7fdc35321800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:01.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=0 2024-05-01T12:57:01.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=1 2024-05-01T12:57:01.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=2 2024-05-01T12:57:01.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=3 2024-05-01T12:57:01.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=4 2024-05-01T12:57:01.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=5 2024-05-01T12:57:01.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.614+0000 7fdc35321800 20 add_watcher() i=6 2024-05-01T12:57:01.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.615+0000 7fdc35321800 20 add_watcher() i=7 2024-05-01T12:57:01.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.615+0000 7fdc35321800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:01.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.616+0000 7fdae77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdc35321800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdc35321800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdc35321800 5 note: GC not initialized 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdae67e4640 20 reqs_thread_entry: start 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdc35321800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fdc35321800 20 Filter name: none 2024-05-01T12:57:01.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.617+0000 7fda57fff640 20 reqs_thread_entry: start 2024-05-01T12:57:01.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.618+0000 7fdc35321800 20 rados->read ofs=0 len=0 2024-05-01T12:57:01.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.618+0000 7fdc35321800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.619+0000 7fdae77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.622+0000 7fdc35321800 20 remove_watcher() i=0 2024-05-01T12:57:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.622+0000 7fdc35321800 2 removed watcher, disabling cache 2024-05-01T12:57:01.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.623+0000 7fdc35321800 20 remove_watcher() i=1 2024-05-01T12:57:01.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.624+0000 7fdc35321800 20 remove_watcher() i=2 2024-05-01T12:57:01.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.625+0000 7fdc35321800 20 remove_watcher() i=3 2024-05-01T12:57:01.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.626+0000 7fdc35321800 20 remove_watcher() i=4 2024-05-01T12:57:01.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.627+0000 7fdc35321800 20 remove_watcher() i=5 2024-05-01T12:57:01.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.628+0000 7fdc35321800 20 remove_watcher() i=6 2024-05-01T12:57:01.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:01.629+0000 7fdc35321800 20 remove_watcher() i=7 2024-05-01T12:57:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.725+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.727+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.727+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.728+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.728+0000 7f0e86b35800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.728+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.728+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.728+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.729+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:06.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.729+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:06.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:06.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:06.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.730+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.731+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:06.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.731+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 zone MyZone found 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:06.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:06.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 started sync module instance, tier type = 2024-05-01T12:57:06.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.732+0000 7f0e86b35800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=0 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=1 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=2 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=3 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=4 2024-05-01T12:57:06.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=5 2024-05-01T12:57:06.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.743+0000 7f0e86b35800 20 add_watcher() i=6 2024-05-01T12:57:06.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.744+0000 7f0e86b35800 20 add_watcher() i=7 2024-05-01T12:57:06.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.744+0000 7f0e86b35800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:06.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0d38fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0e86b35800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0e86b35800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0e86b35800 5 note: GC not initialized 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0cb37fe640 20 reqs_thread_entry: start 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0e86b35800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0e86b35800 20 Filter name: none 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.746+0000 7f0cb17fa640 20 reqs_thread_entry: start 2024-05-01T12:57:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.747+0000 7f0e86b35800 20 rados->read ofs=0 len=0 2024-05-01T12:57:06.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.747+0000 7f0e86b35800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:06.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.748+0000 7f0d38fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:06.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.750+0000 7f0e86b35800 20 remove_watcher() i=0 2024-05-01T12:57:06.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.750+0000 7f0e86b35800 2 removed watcher, disabling cache 2024-05-01T12:57:06.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.751+0000 7f0e86b35800 20 remove_watcher() i=1 2024-05-01T12:57:06.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.752+0000 7f0e86b35800 20 remove_watcher() i=2 2024-05-01T12:57:06.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.753+0000 7f0e86b35800 20 remove_watcher() i=3 2024-05-01T12:57:06.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.754+0000 7f0e86b35800 20 remove_watcher() i=4 2024-05-01T12:57:06.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.756+0000 7f0e86b35800 20 remove_watcher() i=5 2024-05-01T12:57:06.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.757+0000 7f0e86b35800 20 remove_watcher() i=6 2024-05-01T12:57:06.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:06.758+0000 7f0e86b35800 20 remove_watcher() i=7 2024-05-01T12:57:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.850+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.852+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.852+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.852+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.852+0000 7f78adb38800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.852+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.853+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.853+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.853+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.853+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.854+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.854+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.854+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.854+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.855+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.855+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.855+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.855+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 zone MyZone found 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.856+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.857+0000 7f78adb38800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.857+0000 7f78adb38800 20 started sync module instance, tier type = 2024-05-01T12:57:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.857+0000 7f78adb38800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=0 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=1 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=2 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=3 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=4 2024-05-01T12:57:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=5 2024-05-01T12:57:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.868+0000 7f78adb38800 20 add_watcher() i=6 2024-05-01T12:57:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.869+0000 7f78adb38800 20 add_watcher() i=7 2024-05-01T12:57:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.869+0000 7f78adb38800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f775b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f78adb38800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f78adb38800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f78adb38800 5 note: GC not initialized 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f775a7fc640 20 reqs_thread_entry: start 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.871+0000 7f78adb38800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.872+0000 7f78adb38800 20 Filter name: none 2024-05-01T12:57:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.872+0000 7f76cbfff640 20 reqs_thread_entry: start 2024-05-01T12:57:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.872+0000 7f78adb38800 20 rados->read ofs=0 len=0 2024-05-01T12:57:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.872+0000 7f78adb38800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.874+0000 7f775b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.876+0000 7f78adb38800 20 remove_watcher() i=0 2024-05-01T12:57:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.876+0000 7f78adb38800 2 removed watcher, disabling cache 2024-05-01T12:57:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.877+0000 7f78adb38800 20 remove_watcher() i=1 2024-05-01T12:57:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.878+0000 7f78adb38800 20 remove_watcher() i=2 2024-05-01T12:57:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.879+0000 7f78adb38800 20 remove_watcher() i=3 2024-05-01T12:57:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.880+0000 7f78adb38800 20 remove_watcher() i=4 2024-05-01T12:57:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.881+0000 7f78adb38800 20 remove_watcher() i=5 2024-05-01T12:57:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.882+0000 7f78adb38800 20 remove_watcher() i=6 2024-05-01T12:57:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:11.883+0000 7f78adb38800 20 remove_watcher() i=7 2024-05-01T12:57:16.940 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:16.940 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.977+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.978+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:16.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.978+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.979+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.979+0000 7f6788946800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.979+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.979+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.979+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.980+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.980+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:16.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:16.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:16.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.981+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.982+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.982+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 zone MyZone found 2024-05-01T12:57:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:16.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:16.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 started sync module instance, tier type = 2024-05-01T12:57:16.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.983+0000 7f6788946800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=0 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=1 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=2 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=3 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=4 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=5 2024-05-01T12:57:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.997+0000 7f6788946800 20 add_watcher() i=6 2024-05-01T12:57:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.998+0000 7f6788946800 20 add_watcher() i=7 2024-05-01T12:57:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:16.998+0000 7f6788946800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.000+0000 7f663cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.000+0000 7f6788946800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.000+0000 7f6788946800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.000+0000 7f6788946800 5 note: GC not initialized 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f65b77fe640 20 reqs_thread_entry: start 2024-05-01T12:57:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f6788946800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f6788946800 20 Filter name: none 2024-05-01T12:57:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f65b57fa640 20 reqs_thread_entry: start 2024-05-01T12:57:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f6788946800 20 rados->read ofs=0 len=0 2024-05-01T12:57:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.001+0000 7f6788946800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.003+0000 7f663cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.005+0000 7f6788946800 20 remove_watcher() i=0 2024-05-01T12:57:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.005+0000 7f6788946800 2 removed watcher, disabling cache 2024-05-01T12:57:17.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.006+0000 7f6788946800 20 remove_watcher() i=1 2024-05-01T12:57:17.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.007+0000 7f6788946800 20 remove_watcher() i=2 2024-05-01T12:57:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.008+0000 7f6788946800 20 remove_watcher() i=3 2024-05-01T12:57:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.009+0000 7f6788946800 20 remove_watcher() i=4 2024-05-01T12:57:17.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.010+0000 7f6788946800 20 remove_watcher() i=5 2024-05-01T12:57:17.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.012+0000 7f6788946800 20 remove_watcher() i=6 2024-05-01T12:57:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:17.013+0000 7f6788946800 20 remove_watcher() i=7 2024-05-01T12:57:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:22.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.110+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.112+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:22.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.112+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.113+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:22.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.113+0000 7fec1a813800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.113+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.113+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.113+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.114+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.114+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.115+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.116+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.116+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 zone MyZone found 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 started sync module instance, tier type = 2024-05-01T12:57:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.117+0000 7fec1a813800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=0 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=1 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=2 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=3 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=4 2024-05-01T12:57:22.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.129+0000 7fec1a813800 20 add_watcher() i=5 2024-05-01T12:57:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.130+0000 7fec1a813800 20 add_watcher() i=6 2024-05-01T12:57:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.130+0000 7fec1a813800 20 add_watcher() i=7 2024-05-01T12:57:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.130+0000 7fec1a813800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.132+0000 7feac77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.132+0000 7fec1a813800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.132+0000 7fec1a813800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.132+0000 7fec1a813800 5 note: GC not initialized 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7feac67fc640 20 reqs_thread_entry: start 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7fec1a813800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7fec1a813800 20 Filter name: none 2024-05-01T12:57:22.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7fea37fff640 20 reqs_thread_entry: start 2024-05-01T12:57:22.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7fec1a813800 20 rados->read ofs=0 len=0 2024-05-01T12:57:22.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.133+0000 7fec1a813800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:22.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.135+0000 7feac77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:22.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.137+0000 7fec1a813800 20 remove_watcher() i=0 2024-05-01T12:57:22.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.137+0000 7fec1a813800 2 removed watcher, disabling cache 2024-05-01T12:57:22.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.138+0000 7fec1a813800 20 remove_watcher() i=1 2024-05-01T12:57:22.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.139+0000 7fec1a813800 20 remove_watcher() i=2 2024-05-01T12:57:22.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.140+0000 7fec1a813800 20 remove_watcher() i=3 2024-05-01T12:57:22.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.141+0000 7fec1a813800 20 remove_watcher() i=4 2024-05-01T12:57:22.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.142+0000 7fec1a813800 20 remove_watcher() i=5 2024-05-01T12:57:22.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.144+0000 7fec1a813800 20 remove_watcher() i=6 2024-05-01T12:57:22.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:22.145+0000 7fec1a813800 20 remove_watcher() i=7 2024-05-01T12:57:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:27.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.241+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.243+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:27.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.243+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.244+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.244+0000 7fe561f33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.244+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.244+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.244+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.245+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.245+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:27.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:27.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:27.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.246+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.247+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:27.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.247+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:27.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 zone MyZone found 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:27.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.248+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.249+0000 7fe561f33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:27.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.249+0000 7fe561f33800 20 started sync module instance, tier type = 2024-05-01T12:57:27.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.249+0000 7fe561f33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=0 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=1 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=2 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=3 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=4 2024-05-01T12:57:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=5 2024-05-01T12:57:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.259+0000 7fe561f33800 20 add_watcher() i=6 2024-05-01T12:57:27.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.263+0000 7fe561f33800 20 add_watcher() i=7 2024-05-01T12:57:27.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.263+0000 7fe561f33800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:27.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.264+0000 7fe40f7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.264+0000 7fe561f33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.264+0000 7fe561f33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.264+0000 7fe561f33800 5 note: GC not initialized 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.264+0000 7fe40e7fc640 20 reqs_thread_entry: start 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.265+0000 7fe561f33800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.265+0000 7fe561f33800 20 Filter name: none 2024-05-01T12:57:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.265+0000 7fe37ffff640 20 reqs_thread_entry: start 2024-05-01T12:57:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.265+0000 7fe561f33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.265+0000 7fe561f33800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.267+0000 7fe40f7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.269+0000 7fe561f33800 20 remove_watcher() i=0 2024-05-01T12:57:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.269+0000 7fe561f33800 2 removed watcher, disabling cache 2024-05-01T12:57:27.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.270+0000 7fe561f33800 20 remove_watcher() i=1 2024-05-01T12:57:27.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.271+0000 7fe561f33800 20 remove_watcher() i=2 2024-05-01T12:57:27.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.272+0000 7fe561f33800 20 remove_watcher() i=3 2024-05-01T12:57:27.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.273+0000 7fe561f33800 20 remove_watcher() i=4 2024-05-01T12:57:27.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.274+0000 7fe561f33800 20 remove_watcher() i=5 2024-05-01T12:57:27.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.275+0000 7fe561f33800 20 remove_watcher() i=6 2024-05-01T12:57:27.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:27.276+0000 7fe561f33800 20 remove_watcher() i=7 2024-05-01T12:57:32.338 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:32.338 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:32.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.374+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.376+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:32.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.376+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.377+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.377+0000 7fbd37687800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.377+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.378+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.378+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.378+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.378+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.379+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.379+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.380+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.380+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.380+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.380+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 zone MyZone found 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.381+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.382+0000 7fbd37687800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.382+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.382+0000 7fbd37687800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.382+0000 7fbd37687800 20 started sync module instance, tier type = 2024-05-01T12:57:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.382+0000 7fbd37687800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=0 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=1 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=2 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=3 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=4 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=5 2024-05-01T12:57:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.393+0000 7fbd37687800 20 add_watcher() i=6 2024-05-01T12:57:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.395+0000 7fbd37687800 20 add_watcher() i=7 2024-05-01T12:57:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.395+0000 7fbd37687800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbbeb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbd37687800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbd37687800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbd37687800 5 note: GC not initialized 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbbea7e4640 20 reqs_thread_entry: start 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.397+0000 7fbd37687800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.398+0000 7fbd37687800 20 Filter name: none 2024-05-01T12:57:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.398+0000 7fbb5bfff640 20 reqs_thread_entry: start 2024-05-01T12:57:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.398+0000 7fbd37687800 20 rados->read ofs=0 len=0 2024-05-01T12:57:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.398+0000 7fbd37687800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.400+0000 7fbbeb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.402+0000 7fbd37687800 20 remove_watcher() i=0 2024-05-01T12:57:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.402+0000 7fbd37687800 2 removed watcher, disabling cache 2024-05-01T12:57:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.403+0000 7fbd37687800 20 remove_watcher() i=1 2024-05-01T12:57:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.404+0000 7fbd37687800 20 remove_watcher() i=2 2024-05-01T12:57:32.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.405+0000 7fbd37687800 20 remove_watcher() i=3 2024-05-01T12:57:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.406+0000 7fbd37687800 20 remove_watcher() i=4 2024-05-01T12:57:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.407+0000 7fbd37687800 20 remove_watcher() i=5 2024-05-01T12:57:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.408+0000 7fbd37687800 20 remove_watcher() i=6 2024-05-01T12:57:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:32.410+0000 7fbd37687800 20 remove_watcher() i=7 2024-05-01T12:57:32.424 INFO:teuthology.orchestra.run.smithi050.stderr:FAIL 2024-05-01T12:57:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:test persistent topic configurations with time_to_live ... ignoring --setuser ceph since I am not root 2024-05-01T12:57:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:53.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.036+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.038+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.038+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.039+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.039+0000 7f11f4f2d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:53.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.039+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.039+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.039+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.040+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.040+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.041+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.041+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.041+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.041+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.041+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.042+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.042+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.042+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 zone MyZone found 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.043+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.044+0000 7f11f4f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.044+0000 7f11f4f2d800 20 started sync module instance, tier type = 2024-05-01T12:57:53.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.044+0000 7f11f4f2d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=0 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=1 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=2 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=3 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=4 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=5 2024-05-01T12:57:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.057+0000 7f11f4f2d800 20 add_watcher() i=6 2024-05-01T12:57:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.058+0000 7f11f4f2d800 20 add_watcher() i=7 2024-05-01T12:57:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.058+0000 7f11f4f2d800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:53.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.060+0000 7f10a6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.060+0000 7f11f4f2d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:53.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.060+0000 7f11f4f2d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:53.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.060+0000 7f11f4f2d800 5 note: GC not initialized 2024-05-01T12:57:53.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.061+0000 7f10a5fe3640 20 reqs_thread_entry: start 2024-05-01T12:57:53.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.061+0000 7f11f4f2d800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.061+0000 7f11f4f2d800 20 Filter name: none 2024-05-01T12:57:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.061+0000 7f10177fe640 20 reqs_thread_entry: start 2024-05-01T12:57:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.061+0000 7f11f4f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.062+0000 7f11f4f2d800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:53.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.063+0000 7f10a6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.065+0000 7f11f4f2d800 20 remove_watcher() i=0 2024-05-01T12:57:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.065+0000 7f11f4f2d800 2 removed watcher, disabling cache 2024-05-01T12:57:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.066+0000 7f11f4f2d800 20 remove_watcher() i=1 2024-05-01T12:57:53.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.068+0000 7f11f4f2d800 20 remove_watcher() i=2 2024-05-01T12:57:53.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.068+0000 7f11f4f2d800 20 remove_watcher() i=3 2024-05-01T12:57:53.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.070+0000 7f11f4f2d800 20 remove_watcher() i=4 2024-05-01T12:57:53.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.071+0000 7f11f4f2d800 20 remove_watcher() i=5 2024-05-01T12:57:53.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.072+0000 7f11f4f2d800 20 remove_watcher() i=6 2024-05-01T12:57:53.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.075+0000 7f11f4f2d800 20 remove_watcher() i=7 2024-05-01T12:57:53.127 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:53.128 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.164+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.166+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.166+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.167+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.167+0000 7f2a98d33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:53.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.167+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.167+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.167+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.168+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.168+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.169+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.170+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.170+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 zone MyZone found 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 started sync module instance, tier type = 2024-05-01T12:57:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.171+0000 7f2a98d33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=0 2024-05-01T12:57:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=1 2024-05-01T12:57:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=2 2024-05-01T12:57:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=3 2024-05-01T12:57:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=4 2024-05-01T12:57:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=5 2024-05-01T12:57:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.182+0000 7f2a98d33800 20 add_watcher() i=6 2024-05-01T12:57:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.183+0000 7f2a98d33800 20 add_watcher() i=7 2024-05-01T12:57:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.183+0000 7f2a98d33800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.185+0000 7f294afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2a98d33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2a98d33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2a98d33800 5 note: GC not initialized 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2949fe3640 20 reqs_thread_entry: start 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2a98d33800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f2a98d33800 20 Filter name: none 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.186+0000 7f28bb7fe640 20 reqs_thread_entry: start 2024-05-01T12:57:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.187+0000 7f2a98d33800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.187+0000 7f2a98d33800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.188+0000 7f294afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.190+0000 7f2a98d33800 20 remove_watcher() i=0 2024-05-01T12:57:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.190+0000 7f2a98d33800 2 removed watcher, disabling cache 2024-05-01T12:57:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.192+0000 7f2a98d33800 20 remove_watcher() i=1 2024-05-01T12:57:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.192+0000 7f2a98d33800 20 remove_watcher() i=2 2024-05-01T12:57:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.193+0000 7f2a98d33800 20 remove_watcher() i=3 2024-05-01T12:57:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.194+0000 7f2a98d33800 20 remove_watcher() i=4 2024-05-01T12:57:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.195+0000 7f2a98d33800 20 remove_watcher() i=5 2024-05-01T12:57:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.196+0000 7f2a98d33800 20 remove_watcher() i=6 2024-05-01T12:57:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.198+0000 7f2a98d33800 20 remove_watcher() i=7 2024-05-01T12:57:53.498 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:57:53.498 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:57:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.536+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.537+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.538+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.538+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.538+0000 7f4d23b1f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:57:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.538+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.539+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.539+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.540+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:57:53.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.540+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.540+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:57:53.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.540+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.541+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.541+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.541+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:57:53.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.541+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:57:53.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 zone MyZone found 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 period zonegroup name MyZoneGroup 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.542+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.543+0000 7f4d23b1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.543+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.543+0000 7f4d23b1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:57:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.543+0000 7f4d23b1f800 20 started sync module instance, tier type = 2024-05-01T12:57:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.543+0000 7f4d23b1f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:57:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=0 2024-05-01T12:57:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=1 2024-05-01T12:57:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=2 2024-05-01T12:57:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=3 2024-05-01T12:57:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=4 2024-05-01T12:57:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=5 2024-05-01T12:57:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.559+0000 7f4d23b1f800 20 add_watcher() i=6 2024-05-01T12:57:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.560+0000 7f4d23b1f800 20 add_watcher() i=7 2024-05-01T12:57:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.560+0000 7f4d23b1f800 2 all 8 watchers are set, enabling cache 2024-05-01T12:57:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.562+0000 7f4bd5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.562+0000 7f4d23b1f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:57:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.562+0000 7f4d23b1f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:57:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.562+0000 7f4d23b1f800 5 note: GC not initialized 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.562+0000 7f4bd4fe1640 20 reqs_thread_entry: start 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.563+0000 7f4d23b1f800 20 init_complete bucket index max shards: 11 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.563+0000 7f4d23b1f800 20 Filter name: none 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.563+0000 7f4b4e7fc640 20 reqs_thread_entry: start 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.563+0000 7f4d23b1f800 20 rados->read ofs=0 len=0 2024-05-01T12:57:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.563+0000 7f4d23b1f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:57:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.565+0000 7f4bd5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:57:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.567+0000 7f4d23b1f800 20 remove_watcher() i=0 2024-05-01T12:57:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.567+0000 7f4d23b1f800 2 removed watcher, disabling cache 2024-05-01T12:57:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.568+0000 7f4d23b1f800 20 remove_watcher() i=1 2024-05-01T12:57:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.569+0000 7f4d23b1f800 20 remove_watcher() i=2 2024-05-01T12:57:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.570+0000 7f4d23b1f800 20 remove_watcher() i=3 2024-05-01T12:57:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.571+0000 7f4d23b1f800 20 remove_watcher() i=4 2024-05-01T12:57:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.572+0000 7f4d23b1f800 20 remove_watcher() i=5 2024-05-01T12:57:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.573+0000 7f4d23b1f800 20 remove_watcher() i=6 2024-05-01T12:57:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:57:53.575+0000 7f4d23b1f800 20 remove_watcher() i=7 2024-05-01T12:58:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:58:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:58:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.670+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.672+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.672+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.673+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.673+0000 7fb1caa98800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:58:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.673+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.673+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.673+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.674+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:58:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.674+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.674+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:58:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.674+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.675+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.675+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.675+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.675+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 zone MyZone found 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 period zonegroup name MyZoneGroup 2024-05-01T12:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.676+0000 7fb1caa98800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.677+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.677+0000 7fb1caa98800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.677+0000 7fb1caa98800 20 started sync module instance, tier type = 2024-05-01T12:58:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.677+0000 7fb1caa98800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=0 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=1 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=2 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=3 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=4 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=5 2024-05-01T12:58:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.687+0000 7fb1caa98800 20 add_watcher() i=6 2024-05-01T12:58:33.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.688+0000 7fb1caa98800 20 add_watcher() i=7 2024-05-01T12:58:33.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.688+0000 7fb1caa98800 2 all 8 watchers are set, enabling cache 2024-05-01T12:58:33.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.690+0000 7fb07efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:58:33.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.690+0000 7fb1caa98800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:58:33.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.690+0000 7fb1caa98800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:58:33.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.690+0000 7fb1caa98800 5 note: GC not initialized 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.691+0000 7fb07dfdb640 20 reqs_thread_entry: start 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.691+0000 7fb1caa98800 20 init_complete bucket index max shards: 11 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.691+0000 7fb1caa98800 20 Filter name: none 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.691+0000 7fafef7fe640 20 reqs_thread_entry: start 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.692+0000 7fb1caa98800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.692+0000 7fb1caa98800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:58:33.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.694+0000 7fb07efdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:58:33.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.696+0000 7fb1caa98800 20 remove_watcher() i=0 2024-05-01T12:58:33.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.696+0000 7fb1caa98800 2 removed watcher, disabling cache 2024-05-01T12:58:33.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.697+0000 7fb1caa98800 20 remove_watcher() i=1 2024-05-01T12:58:33.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.698+0000 7fb1caa98800 20 remove_watcher() i=2 2024-05-01T12:58:33.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.699+0000 7fb1caa98800 20 remove_watcher() i=3 2024-05-01T12:58:33.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.700+0000 7fb1caa98800 20 remove_watcher() i=4 2024-05-01T12:58:33.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.701+0000 7fb1caa98800 20 remove_watcher() i=5 2024-05-01T12:58:33.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.702+0000 7fb1caa98800 20 remove_watcher() i=6 2024-05-01T12:58:33.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.703+0000 7fb1caa98800 20 remove_watcher() i=7 2024-05-01T12:58:33.772 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:58:33.772 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:58:33.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.809+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.811+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.811+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.812+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.812+0000 7f7aee935800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:58:33.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.812+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.812+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.812+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.813+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:58:33.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.813+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.813+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:58:33.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.813+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.814+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.814+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.814+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:58:33.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.814+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.815+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:58:33.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.815+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.815+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:58:33.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 zone MyZone found 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 period zonegroup name MyZoneGroup 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 started sync module instance, tier type = 2024-05-01T12:58:33.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.816+0000 7f7aee935800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:58:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=0 2024-05-01T12:58:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=1 2024-05-01T12:58:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=2 2024-05-01T12:58:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=3 2024-05-01T12:58:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=4 2024-05-01T12:58:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=5 2024-05-01T12:58:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.828+0000 7f7aee935800 20 add_watcher() i=6 2024-05-01T12:58:33.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.830+0000 7f7aee935800 20 add_watcher() i=7 2024-05-01T12:58:33.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.830+0000 7f7aee935800 2 all 8 watchers are set, enabling cache 2024-05-01T12:58:33.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.831+0000 7f79a0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:58:33.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f7aee935800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f7aee935800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f7aee935800 5 note: GC not initialized 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f791b7fe640 20 reqs_thread_entry: start 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f7aee935800 20 init_complete bucket index max shards: 11 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f7aee935800 20 Filter name: none 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.832+0000 7f79197fa640 20 reqs_thread_entry: start 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.833+0000 7f7aee935800 20 rados->read ofs=0 len=0 2024-05-01T12:58:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.833+0000 7f7aee935800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:58:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.835+0000 7f79a0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:58:33.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.837+0000 7f7aee935800 20 remove_watcher() i=0 2024-05-01T12:58:33.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.837+0000 7f7aee935800 2 removed watcher, disabling cache 2024-05-01T12:58:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.838+0000 7f7aee935800 20 remove_watcher() i=1 2024-05-01T12:58:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.839+0000 7f7aee935800 20 remove_watcher() i=2 2024-05-01T12:58:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.840+0000 7f7aee935800 20 remove_watcher() i=3 2024-05-01T12:58:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.841+0000 7f7aee935800 20 remove_watcher() i=4 2024-05-01T12:58:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.842+0000 7f7aee935800 20 remove_watcher() i=5 2024-05-01T12:58:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.843+0000 7f7aee935800 20 remove_watcher() i=6 2024-05-01T12:58:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:58:33.845+0000 7f7aee935800 20 remove_watcher() i=7 2024-05-01T12:59:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:59:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:59:13.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.975+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.977+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:13.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.977+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.977+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.977+0000 7fb70a331800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.977+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.978+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.978+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.978+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.978+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.979+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.979+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.979+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:13.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.979+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.980+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:13.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.980+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.980+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:13.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.980+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 20 zone MyZone found 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 20 period zonegroup name MyZoneGroup 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.981+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.982+0000 7fb70a331800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.982+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.982+0000 7fb70a331800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.982+0000 7fb70a331800 20 started sync module instance, tier type = 2024-05-01T12:59:13.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.982+0000 7fb70a331800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=0 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=1 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=2 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=3 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=4 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=5 2024-05-01T12:59:13.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.995+0000 7fb70a331800 20 add_watcher() i=6 2024-05-01T12:59:13.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.996+0000 7fb70a331800 20 add_watcher() i=7 2024-05-01T12:59:13.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.996+0000 7fb70a331800 2 all 8 watchers are set, enabling cache 2024-05-01T12:59:13.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.998+0000 7fb5b77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:13.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb70a331800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb70a331800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb70a331800 5 note: GC not initialized 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb5b67fc640 20 reqs_thread_entry: start 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb70a331800 20 init_complete bucket index max shards: 11 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb70a331800 20 Filter name: none 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:13.999+0000 7fb527fff640 20 reqs_thread_entry: start 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.000+0000 7fb70a331800 20 rados->read ofs=0 len=0 2024-05-01T12:59:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.000+0000 7fb70a331800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:59:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.002+0000 7fb5b77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.004+0000 7fb70a331800 20 remove_watcher() i=0 2024-05-01T12:59:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.004+0000 7fb70a331800 2 removed watcher, disabling cache 2024-05-01T12:59:14.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.005+0000 7fb70a331800 20 remove_watcher() i=1 2024-05-01T12:59:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.006+0000 7fb70a331800 20 remove_watcher() i=2 2024-05-01T12:59:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.007+0000 7fb70a331800 20 remove_watcher() i=3 2024-05-01T12:59:14.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.008+0000 7fb70a331800 20 remove_watcher() i=4 2024-05-01T12:59:14.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.009+0000 7fb70a331800 20 remove_watcher() i=5 2024-05-01T12:59:14.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.011+0000 7fb70a331800 20 remove_watcher() i=6 2024-05-01T12:59:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:14.012+0000 7fb70a331800 20 remove_watcher() i=7 2024-05-01T12:59:14.023 INFO:teuthology.orchestra.run.smithi050.stdout:"topic not found" error is expected 2024-05-01T12:59:14.023 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'GET /ashubx-4?notification=ashubx-4_notif_1 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:OkFsE9oNH1gsIO26G1BTUQ2qO+0=\r\nDate: Wed, 01 May 2024 12:53:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000a50253c3e38bf774-0066323b58-4559-MyZone 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Type: application/xml 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 208 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:53:44 GMT 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-4?notification=ashubx-4_notif_1 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:W5yInBTlEBENUlIImktiPGVpIVU=\r\nDate: Wed, 01 May 2024 12:53:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx0000066d7df9a31d4223f-0066323b58-4559-MyZone 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:53:44 GMT 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-4?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:tq7S0IUztS/jU53CPuesNac6YK8=\r\nDate: Wed, 01 May 2024 12:53:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000003d997165d362bbf7-0066323b58-4559-MyZone 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:53:44 GMT 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'GET /ashubx-5?notification=ashubx-5_notif_1 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:MciJ5XP4xdcdwTtTeTP5+bhkiK4=\r\nDate: Wed, 01 May 2024 12:53:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx0000073fc8b7e4263cd48-0066323b58-4559-MyZone 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Type: application/xml 2024-05-01T12:59:14.025 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 241 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:53:44 GMT 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unknown Event type: ashubx-6_notif - is expected 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Missing required element Id - is expected 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Invalid Topic ARN - is expected 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:An error occurred (NoSuchKey) when calling the PutBucketNotificationConfiguration operation: None - is expected 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:An error occurred (NoSuchBucket) when calling the PutBucketNotificationConfiguration operation: None - is expected 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + http notification is: 103.255295753479 milliseconds 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-7?notification=ashubx-7_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:MR/SUguSM9Gp0hKR5gng5fEL0bU=\r\nDate: Wed, 01 May 2024 12:53:50 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.026 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000005f0a9229e1db2639-0066323b5e-4559-MyZone 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:53:50 GMT 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 23.56100082397461 milliseconds 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + http notification is: 2.7696847915649414 milliseconds 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-8?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:K0k214kVwZEHTBf8SwstNz3n7Vw=\r\nDate: Wed, 01 May 2024 12:54:01 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000003a1a96eae6ea9fd3-0066323b69-4559-MyZone 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:01 GMT 2024-05-01T12:59:14.027 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 5.395936965942383 milliseconds 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + http notification is: 2.9985427856445312 milliseconds 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-9?notification=ashubx-9_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:UmIxcDQU3i/LNX3Wk31cvd4zISI=\r\nDate: Wed, 01 May 2024 12:54:12 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000006c3e74f83fea6ef5-0066323b74-4559-MyZone 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:12 GMT 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 2.3215293884277344 milliseconds 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:total number of objects: 10 2024-05-01T12:59:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-10?notification=ashubx-10_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:fgbAYUV5CJG2Zng2Q6P/YM86MgM=\r\nDate: Wed, 01 May 2024 12:54:17 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000009f844908cd9f26c4-0066323b79-4559-MyZone 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:17 GMT 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 5.283951759338379 milliseconds 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:total number of objects: 0 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-11?notification=ashubx-11_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:wwUhPBZex/X19QW5BKNMVLAOcLA=\r\nDate: Wed, 01 May 2024 12:54:25 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000005294f8d91a685507-0066323b81-4559-MyZone 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:25 GMT 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 1236.3178730010986 milliseconds 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 20s (2 days) for the messages... 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:total number of objects: 0 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-12?notification=ashubx-12_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:DjvgmCfOCaBV/dN+Ko/Hw8M44Es=\r\nDate: Wed, 01 May 2024 12:54:49 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000006f280aca9f1a784b-0066323b99-4559-MyZone 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:49 GMT 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:wait for 5sec for the messages... 2024-05-01T12:59:14.030 INFO:teuthology.orchestra.run.smithi050.stdout:total number of objects: 1 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-13?notification=ashubx-13_notif HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:Q4abqUxnqdwEsHgtamFgAECrJTs=\r\nDate: Wed, 01 May 2024 12:54:54 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx0000041c2ee24c236ce19-0066323b9e-4559-MyZone 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:54:54 GMT 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 23.73265266418457 milliseconds 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + async http notification is: 1.987287998199463 milliseconds 2024-05-01T12:59:14.031 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 24.67331886291504 milliseconds 2024-05-01T12:59:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T12:59:54.632 INFO:teuthology.orchestra.run.smithi050.stderr:test persistent topic configurations with max_retries and retry_sleep_duration ... ignoring --setuser ceph since I am not root 2024-05-01T12:59:54.632 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:59:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.670+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.671+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.671+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.672+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.672+0000 7fd011a62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:59:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.672+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.672+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.672+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.673+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:59:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.673+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.674+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:59:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.674+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.674+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.674+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.675+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.675+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.675+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.675+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 zone MyZone found 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 period zonegroup name MyZoneGroup 2024-05-01T12:59:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.676+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.677+0000 7fd011a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.677+0000 7fd011a62800 20 started sync module instance, tier type = 2024-05-01T12:59:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.677+0000 7fd011a62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=0 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=1 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=2 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=3 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=4 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=5 2024-05-01T12:59:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.687+0000 7fd011a62800 20 add_watcher() i=6 2024-05-01T12:59:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.689+0000 7fd011a62800 20 add_watcher() i=7 2024-05-01T12:59:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.689+0000 7fd011a62800 2 all 8 watchers are set, enabling cache 2024-05-01T12:59:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fcec5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fd011a62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fd011a62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fd011a62800 5 note: GC not initialized 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fcec4fe1640 20 reqs_thread_entry: start 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fd011a62800 20 init_complete bucket index max shards: 11 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.691+0000 7fd011a62800 20 Filter name: none 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.692+0000 7fce3e7fc640 20 reqs_thread_entry: start 2024-05-01T12:59:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.692+0000 7fd011a62800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.693+0000 7fd011a62800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:59:54.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.694+0000 7fcec5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:54.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.696+0000 7fd011a62800 20 remove_watcher() i=0 2024-05-01T12:59:54.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.696+0000 7fd011a62800 2 removed watcher, disabling cache 2024-05-01T12:59:54.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.697+0000 7fd011a62800 20 remove_watcher() i=1 2024-05-01T12:59:54.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.698+0000 7fd011a62800 20 remove_watcher() i=2 2024-05-01T12:59:54.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.699+0000 7fd011a62800 20 remove_watcher() i=3 2024-05-01T12:59:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.700+0000 7fd011a62800 20 remove_watcher() i=4 2024-05-01T12:59:54.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.701+0000 7fd011a62800 20 remove_watcher() i=5 2024-05-01T12:59:54.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.703+0000 7fd011a62800 20 remove_watcher() i=6 2024-05-01T12:59:54.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.704+0000 7fd011a62800 20 remove_watcher() i=7 2024-05-01T12:59:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:59:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:59:54.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.794+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.795+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.795+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.796+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.796+0000 7f1fe5f2d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:59:54.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.796+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.797+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.797+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.797+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:59:54.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.797+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:59:54.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:54.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.798+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.799+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:54.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.799+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 zone MyZone found 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 period zonegroup name MyZoneGroup 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:54.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.800+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.801+0000 7f1fe5f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.801+0000 7f1fe5f2d800 20 started sync module instance, tier type = 2024-05-01T12:59:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.801+0000 7f1fe5f2d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:59:54.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=0 2024-05-01T12:59:54.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=1 2024-05-01T12:59:54.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=2 2024-05-01T12:59:54.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=3 2024-05-01T12:59:54.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=4 2024-05-01T12:59:54.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=5 2024-05-01T12:59:54.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.812+0000 7f1fe5f2d800 20 add_watcher() i=6 2024-05-01T12:59:54.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.813+0000 7f1fe5f2d800 20 add_watcher() i=7 2024-05-01T12:59:54.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.813+0000 7f1fe5f2d800 2 all 8 watchers are set, enabling cache 2024-05-01T12:59:54.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.815+0000 7f1e937fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 5 note: GC not initialized 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1e927fc640 20 reqs_thread_entry: start 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 20 init_complete bucket index max shards: 11 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 20 Filter name: none 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1e03fff640 20 reqs_thread_entry: start 2024-05-01T12:59:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.816+0000 7f1fe5f2d800 20 rados->read ofs=0 len=0 2024-05-01T12:59:54.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.817+0000 7f1fe5f2d800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:59:54.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.818+0000 7f1e937fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:54.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.820+0000 7f1fe5f2d800 20 remove_watcher() i=0 2024-05-01T12:59:54.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.820+0000 7f1fe5f2d800 2 removed watcher, disabling cache 2024-05-01T12:59:54.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.822+0000 7f1fe5f2d800 20 remove_watcher() i=1 2024-05-01T12:59:54.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.823+0000 7f1fe5f2d800 20 remove_watcher() i=2 2024-05-01T12:59:54.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.824+0000 7f1fe5f2d800 20 remove_watcher() i=3 2024-05-01T12:59:54.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.825+0000 7f1fe5f2d800 20 remove_watcher() i=4 2024-05-01T12:59:54.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.826+0000 7f1fe5f2d800 20 remove_watcher() i=5 2024-05-01T12:59:54.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.827+0000 7f1fe5f2d800 20 remove_watcher() i=6 2024-05-01T12:59:54.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:54.829+0000 7f1fe5f2d800 20 remove_watcher() i=7 2024-05-01T12:59:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T12:59:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T12:59:55.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.186+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.188+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:55.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.188+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.188+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.188+0000 7f83d4133800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.188+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.189+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.189+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.189+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T12:59:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.189+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.190+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T12:59:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.190+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.190+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.190+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.191+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T12:59:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.191+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.191+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T12:59:55.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.191+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 zone MyZone found 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 period zonegroup name MyZoneGroup 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 using current period zonegroup MyZoneGroup 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.192+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.193+0000 7f83d4133800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T12:59:55.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.193+0000 7f83d4133800 20 started sync module instance, tier type = 2024-05-01T12:59:55.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.193+0000 7f83d4133800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=0 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=1 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=2 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=3 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=4 2024-05-01T12:59:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.204+0000 7f83d4133800 20 add_watcher() i=5 2024-05-01T12:59:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.205+0000 7f83d4133800 20 add_watcher() i=6 2024-05-01T12:59:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.206+0000 7f83d4133800 20 add_watcher() i=7 2024-05-01T12:59:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.206+0000 7f83d4133800 2 all 8 watchers are set, enabling cache 2024-05-01T12:59:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.208+0000 7f8285fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.208+0000 7f83d4133800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T12:59:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.208+0000 7f83d4133800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.208+0000 7f83d4133800 5 note: GC not initialized 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.209+0000 7f8284fe1640 20 reqs_thread_entry: start 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.209+0000 7f83d4133800 20 init_complete bucket index max shards: 11 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.209+0000 7f83d4133800 20 Filter name: none 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.209+0000 7f81fe7fc640 20 reqs_thread_entry: start 2024-05-01T12:59:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.209+0000 7f83d4133800 20 rados->read ofs=0 len=0 2024-05-01T12:59:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.210+0000 7f83d4133800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T12:59:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.211+0000 7f8285fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T12:59:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.213+0000 7f83d4133800 20 remove_watcher() i=0 2024-05-01T12:59:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.213+0000 7f83d4133800 2 removed watcher, disabling cache 2024-05-01T12:59:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.214+0000 7f83d4133800 20 remove_watcher() i=1 2024-05-01T12:59:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.216+0000 7f83d4133800 20 remove_watcher() i=2 2024-05-01T12:59:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.217+0000 7f83d4133800 20 remove_watcher() i=3 2024-05-01T12:59:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.218+0000 7f83d4133800 20 remove_watcher() i=4 2024-05-01T12:59:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.219+0000 7f83d4133800 20 remove_watcher() i=5 2024-05-01T12:59:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.220+0000 7f83d4133800 20 remove_watcher() i=6 2024-05-01T12:59:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T12:59:55.221+0000 7f83d4133800 20 remove_watcher() i=7 2024-05-01T13:00:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:00:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:00:35.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.333+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.335+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.335+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.336+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.336+0000 7f8c15416800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.336+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.336+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.336+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.337+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:00:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.337+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:00:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.338+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.339+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.339+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:00:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 zone MyZone found 2024-05-01T13:00:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:00:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 period zonegroup name MyZoneGroup 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:00:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:00:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 started sync module instance, tier type = 2024-05-01T13:00:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.340+0000 7f8c15416800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:00:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=0 2024-05-01T13:00:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=1 2024-05-01T13:00:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=2 2024-05-01T13:00:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=3 2024-05-01T13:00:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=4 2024-05-01T13:00:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=5 2024-05-01T13:00:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.351+0000 7f8c15416800 20 add_watcher() i=6 2024-05-01T13:00:35.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.354+0000 7f8c15416800 20 add_watcher() i=7 2024-05-01T13:00:35.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.354+0000 7f8c15416800 2 all 8 watchers are set, enabling cache 2024-05-01T13:00:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.355+0000 7f8ac6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8c15416800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8c15416800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8c15416800 5 note: GC not initialized 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8ac5fe3640 20 reqs_thread_entry: start 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8c15416800 20 init_complete bucket index max shards: 11 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8c15416800 20 Filter name: none 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.356+0000 7f8a377fe640 20 reqs_thread_entry: start 2024-05-01T13:00:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.357+0000 7f8c15416800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.357+0000 7f8c15416800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:00:35.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.358+0000 7f8ac6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:00:35.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.360+0000 7f8c15416800 20 remove_watcher() i=0 2024-05-01T13:00:35.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.360+0000 7f8c15416800 2 removed watcher, disabling cache 2024-05-01T13:00:35.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.361+0000 7f8c15416800 20 remove_watcher() i=1 2024-05-01T13:00:35.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.362+0000 7f8c15416800 20 remove_watcher() i=2 2024-05-01T13:00:35.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.363+0000 7f8c15416800 20 remove_watcher() i=3 2024-05-01T13:00:35.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.365+0000 7f8c15416800 20 remove_watcher() i=4 2024-05-01T13:00:35.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.366+0000 7f8c15416800 20 remove_watcher() i=5 2024-05-01T13:00:35.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.367+0000 7f8c15416800 20 remove_watcher() i=6 2024-05-01T13:00:35.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.368+0000 7f8c15416800 20 remove_watcher() i=7 2024-05-01T13:00:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:00:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:00:35.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.478+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.480+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.480+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.481+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.481+0000 7f88ed32d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:00:35.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.481+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.481+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.481+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.482+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:00:35.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.482+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.482+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:00:35.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.483+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.483+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.483+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.483+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:00:35.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.483+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.484+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:00:35.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.484+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:00:35.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 zone MyZone found 2024-05-01T13:00:35.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:00:35.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:00:35.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:00:35.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 started sync module instance, tier type = 2024-05-01T13:00:35.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.485+0000 7f88ed32d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=0 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=1 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=2 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=3 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=4 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=5 2024-05-01T13:00:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.497+0000 7f88ed32d800 20 add_watcher() i=6 2024-05-01T13:00:35.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.498+0000 7f88ed32d800 20 add_watcher() i=7 2024-05-01T13:00:35.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.498+0000 7f88ed32d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:00:35.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.500+0000 7f879efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 5 note: GC not initialized 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f879dfe3640 20 reqs_thread_entry: start 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 20 init_complete bucket index max shards: 11 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 20 Filter name: none 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f870f7fe640 20 reqs_thread_entry: start 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.501+0000 7f88ed32d800 20 rados->read ofs=0 len=0 2024-05-01T13:00:35.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.502+0000 7f88ed32d800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:00:35.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.503+0000 7f879efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:00:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.506+0000 7f88ed32d800 20 remove_watcher() i=0 2024-05-01T13:00:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.506+0000 7f88ed32d800 2 removed watcher, disabling cache 2024-05-01T13:00:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.507+0000 7f88ed32d800 20 remove_watcher() i=1 2024-05-01T13:00:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.508+0000 7f88ed32d800 20 remove_watcher() i=2 2024-05-01T13:00:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.509+0000 7f88ed32d800 20 remove_watcher() i=3 2024-05-01T13:00:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.510+0000 7f88ed32d800 20 remove_watcher() i=4 2024-05-01T13:00:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.512+0000 7f88ed32d800 20 remove_watcher() i=5 2024-05-01T13:00:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.513+0000 7f88ed32d800 20 remove_watcher() i=6 2024-05-01T13:00:35.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:00:35.515+0000 7f88ed32d800 20 remove_watcher() i=7 2024-05-01T13:01:15.609 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:01:15.610 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:15.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.646+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.648+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:15.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.648+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:15.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:15.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:15.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:15.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.649+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.650+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.650+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.650+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.650+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.651+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.651+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.651+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.651+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 zone MyZone found 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.652+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.653+0000 7f7c15e98800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.653+0000 7f7c15e98800 20 started sync module instance, tier type = 2024-05-01T13:01:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.653+0000 7f7c15e98800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=0 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=1 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=2 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=3 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=4 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=5 2024-05-01T13:01:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.664+0000 7f7c15e98800 20 add_watcher() i=6 2024-05-01T13:01:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.665+0000 7f7c15e98800 20 add_watcher() i=7 2024-05-01T13:01:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.665+0000 7f7c15e98800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:15.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7ac9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7c15e98800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7c15e98800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7c15e98800 5 note: GC not initialized 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7ac8fe1640 20 reqs_thread_entry: start 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.667+0000 7f7c15e98800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.668+0000 7f7c15e98800 20 Filter name: none 2024-05-01T13:01:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.668+0000 7f7a427fc640 20 reqs_thread_entry: start 2024-05-01T13:01:15.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.668+0000 7f7c15e98800 20 rados->read ofs=0 len=0 2024-05-01T13:01:15.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.668+0000 7f7c15e98800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:01:15.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.670+0000 7f7ac9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.672+0000 7f7c15e98800 20 remove_watcher() i=0 2024-05-01T13:01:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.672+0000 7f7c15e98800 2 removed watcher, disabling cache 2024-05-01T13:01:15.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.673+0000 7f7c15e98800 20 remove_watcher() i=1 2024-05-01T13:01:15.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.674+0000 7f7c15e98800 20 remove_watcher() i=2 2024-05-01T13:01:15.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.675+0000 7f7c15e98800 20 remove_watcher() i=3 2024-05-01T13:01:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.676+0000 7f7c15e98800 20 remove_watcher() i=4 2024-05-01T13:01:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.677+0000 7f7c15e98800 20 remove_watcher() i=5 2024-05-01T13:01:15.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.678+0000 7f7c15e98800 20 remove_watcher() i=6 2024-05-01T13:01:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:15.679+0000 7f7c15e98800 20 remove_watcher() i=7 2024-05-01T13:01:35.756 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:01:36.129 INFO:teuthology.orchestra.run.smithi050.stderr:test pushing persistent notification when one of the endpoints has error ... ignoring --setuser ceph since I am not root 2024-05-01T13:01:36.130 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:36.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.166+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.168+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:36.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.168+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.169+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.169+0000 7fe49723e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.169+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.169+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.169+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.170+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:36.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.170+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.171+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.172+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.172+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:36.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 zone MyZone found 2024-05-01T13:01:36.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 started sync module instance, tier type = 2024-05-01T13:01:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.173+0000 7fe49723e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=0 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=1 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=2 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=3 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=4 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=5 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.184+0000 7fe49723e800 20 add_watcher() i=6 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.185+0000 7fe49723e800 20 add_watcher() i=7 2024-05-01T13:01:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.185+0000 7fe49723e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:36.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.187+0000 7fe33b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.187+0000 7fe49723e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.187+0000 7fe49723e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.187+0000 7fe49723e800 5 note: GC not initialized 2024-05-01T13:01:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.188+0000 7fe33a7fc640 20 reqs_thread_entry: start 2024-05-01T13:01:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.188+0000 7fe49723e800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.188+0000 7fe49723e800 20 Filter name: none 2024-05-01T13:01:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.188+0000 7fe2b3fff640 20 reqs_thread_entry: start 2024-05-01T13:01:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.188+0000 7fe49723e800 20 rados->read ofs=0 len=0 2024-05-01T13:01:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.189+0000 7fe49723e800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:01:36.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.190+0000 7fe33b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:36.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.193+0000 7fe49723e800 20 remove_watcher() i=0 2024-05-01T13:01:36.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.193+0000 7fe49723e800 2 removed watcher, disabling cache 2024-05-01T13:01:36.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.195+0000 7fe49723e800 20 remove_watcher() i=1 2024-05-01T13:01:36.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.197+0000 7fe49723e800 20 remove_watcher() i=2 2024-05-01T13:01:36.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.199+0000 7fe49723e800 20 remove_watcher() i=3 2024-05-01T13:01:36.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.201+0000 7fe49723e800 20 remove_watcher() i=4 2024-05-01T13:01:36.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.203+0000 7fe49723e800 20 remove_watcher() i=5 2024-05-01T13:01:36.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.204+0000 7fe49723e800 20 remove_watcher() i=6 2024-05-01T13:01:36.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:36.206+0000 7fe49723e800 20 remove_watcher() i=7 2024-05-01T13:01:41.265 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:01:41.265 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.303+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.305+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:41.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.305+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.306+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.306+0000 7f0cfb810800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.306+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.307+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.307+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.307+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.307+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.308+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.308+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.308+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.308+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.309+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.309+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.309+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.309+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 zone MyZone found 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:41.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 started sync module instance, tier type = 2024-05-01T13:01:41.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.310+0000 7f0cfb810800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:41.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=0 2024-05-01T13:01:41.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=1 2024-05-01T13:01:41.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=2 2024-05-01T13:01:41.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=3 2024-05-01T13:01:41.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=4 2024-05-01T13:01:41.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=5 2024-05-01T13:01:41.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.321+0000 7f0cfb810800 20 add_watcher() i=6 2024-05-01T13:01:41.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.323+0000 7f0cfb810800 20 add_watcher() i=7 2024-05-01T13:01:41.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.323+0000 7f0cfb810800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:41.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0bad7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:41.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0cfb810800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:41.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0cfb810800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:41.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0cfb810800 5 note: GC not initialized 2024-05-01T13:01:41.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0b27fff640 20 reqs_thread_entry: start 2024-05-01T13:01:41.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0cfb810800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:41.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0cfb810800 20 Filter name: none 2024-05-01T13:01:41.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.325+0000 7f0b25ffb640 20 reqs_thread_entry: start 2024-05-01T13:01:41.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.326+0000 7f0cfb810800 20 rados->read ofs=0 len=0 2024-05-01T13:01:41.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.326+0000 7f0cfb810800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:01:41.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.328+0000 7f0bad7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.330+0000 7f0cfb810800 20 remove_watcher() i=0 2024-05-01T13:01:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.330+0000 7f0cfb810800 2 removed watcher, disabling cache 2024-05-01T13:01:41.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.331+0000 7f0cfb810800 20 remove_watcher() i=1 2024-05-01T13:01:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.332+0000 7f0cfb810800 20 remove_watcher() i=2 2024-05-01T13:01:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.333+0000 7f0cfb810800 20 remove_watcher() i=3 2024-05-01T13:01:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.334+0000 7f0cfb810800 20 remove_watcher() i=4 2024-05-01T13:01:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.335+0000 7f0cfb810800 20 remove_watcher() i=5 2024-05-01T13:01:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.336+0000 7f0cfb810800 20 remove_watcher() i=6 2024-05-01T13:01:41.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:41.337+0000 7f0cfb810800 20 remove_watcher() i=7 2024-05-01T13:01:46.396 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:01:46.396 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.433+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.435+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.435+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.436+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:46.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.436+0000 7ff75f298800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:46.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.436+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.436+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:46.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.436+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.437+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:46.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.437+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.437+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.437+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.438+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.438+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.438+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:46.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.438+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:46.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:46.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 zone MyZone found 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.439+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.440+0000 7ff75f298800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.440+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.440+0000 7ff75f298800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.440+0000 7ff75f298800 20 started sync module instance, tier type = 2024-05-01T13:01:46.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.440+0000 7ff75f298800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=0 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=1 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=2 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=3 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=4 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=5 2024-05-01T13:01:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.451+0000 7ff75f298800 20 add_watcher() i=6 2024-05-01T13:01:46.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.452+0000 7ff75f298800 20 add_watcher() i=7 2024-05-01T13:01:46.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.452+0000 7ff75f298800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:46.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.453+0000 7ff612fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 5 note: GC not initialized 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff611fe3640 20 reqs_thread_entry: start 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 20 Filter name: none 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff5837fe640 20 reqs_thread_entry: start 2024-05-01T13:01:46.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.454+0000 7ff75f298800 20 rados->read ofs=0 len=0 2024-05-01T13:01:46.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.455+0000 7ff75f298800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:01:46.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.456+0000 7ff612fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:46.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.458+0000 7ff75f298800 20 remove_watcher() i=0 2024-05-01T13:01:46.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.458+0000 7ff75f298800 2 removed watcher, disabling cache 2024-05-01T13:01:46.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.459+0000 7ff75f298800 20 remove_watcher() i=1 2024-05-01T13:01:46.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.460+0000 7ff75f298800 20 remove_watcher() i=2 2024-05-01T13:01:46.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.461+0000 7ff75f298800 20 remove_watcher() i=3 2024-05-01T13:01:46.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.462+0000 7ff75f298800 20 remove_watcher() i=4 2024-05-01T13:01:46.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.463+0000 7ff75f298800 20 remove_watcher() i=5 2024-05-01T13:01:46.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.464+0000 7ff75f298800 20 remove_watcher() i=6 2024-05-01T13:01:46.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:46.465+0000 7ff75f298800 20 remove_watcher() i=7 2024-05-01T13:01:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:01:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.563+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.565+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:51.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.565+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.565+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.565+0000 7f5f1c9af800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.565+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.566+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:51.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.566+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.566+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.566+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.567+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.567+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.567+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.567+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.568+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.568+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.569+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:51.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.569+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.569+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 zone MyZone found 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 started sync module instance, tier type = 2024-05-01T13:01:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.570+0000 7f5f1c9af800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=0 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=1 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=2 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=3 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=4 2024-05-01T13:01:51.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=5 2024-05-01T13:01:51.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.581+0000 7f5f1c9af800 20 add_watcher() i=6 2024-05-01T13:01:51.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.582+0000 7f5f1c9af800 20 add_watcher() i=7 2024-05-01T13:01:51.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.582+0000 7f5f1c9af800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:51.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5dd0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5f1c9af800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5f1c9af800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5f1c9af800 5 note: GC not initialized 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5d4b7fe640 20 reqs_thread_entry: start 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5f1c9af800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5f1c9af800 20 Filter name: none 2024-05-01T13:01:51.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.584+0000 7f5d497fa640 20 reqs_thread_entry: start 2024-05-01T13:01:51.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.585+0000 7f5f1c9af800 20 rados->read ofs=0 len=0 2024-05-01T13:01:51.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.585+0000 7f5f1c9af800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:01:51.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.587+0000 7f5dd0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:51.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.589+0000 7f5f1c9af800 20 remove_watcher() i=0 2024-05-01T13:01:51.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.589+0000 7f5f1c9af800 2 removed watcher, disabling cache 2024-05-01T13:01:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.591+0000 7f5f1c9af800 20 remove_watcher() i=1 2024-05-01T13:01:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.592+0000 7f5f1c9af800 20 remove_watcher() i=2 2024-05-01T13:01:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.593+0000 7f5f1c9af800 20 remove_watcher() i=3 2024-05-01T13:01:51.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.594+0000 7f5f1c9af800 20 remove_watcher() i=4 2024-05-01T13:01:51.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.595+0000 7f5f1c9af800 20 remove_watcher() i=5 2024-05-01T13:01:51.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.596+0000 7f5f1c9af800 20 remove_watcher() i=6 2024-05-01T13:01:51.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:51.597+0000 7f5f1c9af800 20 remove_watcher() i=7 2024-05-01T13:01:56.657 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:01:56.657 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:01:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.694+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.696+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.696+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.696+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.696+0000 7f68a6c36800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.696+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.697+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.697+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.697+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.697+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.698+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.698+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.698+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.698+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.699+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.699+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 zone MyZone found 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:01:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 period zonegroup name MyZoneGroup 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.700+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.701+0000 7f68a6c36800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.701+0000 7f68a6c36800 20 started sync module instance, tier type = 2024-05-01T13:01:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.701+0000 7f68a6c36800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=0 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=1 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=2 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=3 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=4 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=5 2024-05-01T13:01:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.712+0000 7f68a6c36800 20 add_watcher() i=6 2024-05-01T13:01:56.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.713+0000 7f68a6c36800 20 add_watcher() i=7 2024-05-01T13:01:56.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.713+0000 7f68a6c36800 2 all 8 watchers are set, enabling cache 2024-05-01T13:01:56.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.715+0000 7f675a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 5 note: GC not initialized 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f67597e2640 20 reqs_thread_entry: start 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 20 init_complete bucket index max shards: 11 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 20 Filter name: none 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f66caffd640 20 reqs_thread_entry: start 2024-05-01T13:01:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.716+0000 7f68a6c36800 20 rados->read ofs=0 len=0 2024-05-01T13:01:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.717+0000 7f68a6c36800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:01:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.719+0000 7f675a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:01:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.721+0000 7f68a6c36800 20 remove_watcher() i=0 2024-05-01T13:01:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.721+0000 7f68a6c36800 2 removed watcher, disabling cache 2024-05-01T13:01:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.722+0000 7f68a6c36800 20 remove_watcher() i=1 2024-05-01T13:01:56.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.723+0000 7f68a6c36800 20 remove_watcher() i=2 2024-05-01T13:01:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.724+0000 7f68a6c36800 20 remove_watcher() i=3 2024-05-01T13:01:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.725+0000 7f68a6c36800 20 remove_watcher() i=4 2024-05-01T13:01:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.726+0000 7f68a6c36800 20 remove_watcher() i=5 2024-05-01T13:01:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.727+0000 7f68a6c36800 20 remove_watcher() i=6 2024-05-01T13:01:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:01:56.728+0000 7f68a6c36800 20 remove_watcher() i=7 2024-05-01T13:02:01.786 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:01.786 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:01.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.823+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.826+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:01.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.826+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.826+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.826+0000 7f7ec340f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.826+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.827+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.827+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.828+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:01.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.828+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.828+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.828+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.829+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.829+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.829+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.829+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:01.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 zone MyZone found 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.830+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.831+0000 7f7ec340f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:01.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.831+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.831+0000 7f7ec340f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:01.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.831+0000 7f7ec340f800 20 started sync module instance, tier type = 2024-05-01T13:02:01.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.831+0000 7f7ec340f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=0 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=1 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=2 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=3 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=4 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=5 2024-05-01T13:02:01.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.843+0000 7f7ec340f800 20 add_watcher() i=6 2024-05-01T13:02:01.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.846+0000 7f7ec340f800 20 add_watcher() i=7 2024-05-01T13:02:01.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.846+0000 7f7ec340f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:01.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7d757e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ec340f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ec340f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ec340f800 5 note: GC not initialized 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ceffff640 20 reqs_thread_entry: start 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ec340f800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7ec340f800 20 Filter name: none 2024-05-01T13:02:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.848+0000 7f7cedffb640 20 reqs_thread_entry: start 2024-05-01T13:02:01.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.849+0000 7f7ec340f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:01.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.849+0000 7f7ec340f800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:01.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.850+0000 7f7d757e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:01.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.853+0000 7f7ec340f800 20 remove_watcher() i=0 2024-05-01T13:02:01.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.853+0000 7f7ec340f800 2 removed watcher, disabling cache 2024-05-01T13:02:01.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.854+0000 7f7ec340f800 20 remove_watcher() i=1 2024-05-01T13:02:01.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.855+0000 7f7ec340f800 20 remove_watcher() i=2 2024-05-01T13:02:01.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.856+0000 7f7ec340f800 20 remove_watcher() i=3 2024-05-01T13:02:01.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.857+0000 7f7ec340f800 20 remove_watcher() i=4 2024-05-01T13:02:01.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.858+0000 7f7ec340f800 20 remove_watcher() i=5 2024-05-01T13:02:01.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.859+0000 7f7ec340f800 20 remove_watcher() i=6 2024-05-01T13:02:01.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:01.861+0000 7f7ec340f800 20 remove_watcher() i=7 2024-05-01T13:02:06.919 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:06.919 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:06.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.957+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.959+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:06.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.959+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.960+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:06.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.960+0000 7ffa0c013800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:06.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.960+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.960+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:06.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.960+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.961+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:06.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.961+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.961+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:06.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.961+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.962+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:06.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.962+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.962+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:06.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.962+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:06.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 zone MyZone found 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.963+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.964+0000 7ffa0c013800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.964+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.964+0000 7ffa0c013800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.964+0000 7ffa0c013800 20 started sync module instance, tier type = 2024-05-01T13:02:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.964+0000 7ffa0c013800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=0 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=1 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=2 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=3 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=4 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=5 2024-05-01T13:02:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.974+0000 7ffa0c013800 20 add_watcher() i=6 2024-05-01T13:02:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.975+0000 7ffa0c013800 20 add_watcher() i=7 2024-05-01T13:02:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.975+0000 7ffa0c013800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.977+0000 7ff8bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:06.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:06.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 5 note: GC not initialized 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ff8bcfe1640 20 reqs_thread_entry: start 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 20 Filter name: none 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ff8367fc640 20 reqs_thread_entry: start 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.978+0000 7ffa0c013800 20 rados->read ofs=0 len=0 2024-05-01T13:02:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.979+0000 7ffa0c013800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:06.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.980+0000 7ff8bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:06.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.982+0000 7ffa0c013800 20 remove_watcher() i=0 2024-05-01T13:02:06.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.982+0000 7ffa0c013800 2 removed watcher, disabling cache 2024-05-01T13:02:06.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.983+0000 7ffa0c013800 20 remove_watcher() i=1 2024-05-01T13:02:06.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.984+0000 7ffa0c013800 20 remove_watcher() i=2 2024-05-01T13:02:06.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.985+0000 7ffa0c013800 20 remove_watcher() i=3 2024-05-01T13:02:06.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.986+0000 7ffa0c013800 20 remove_watcher() i=4 2024-05-01T13:02:06.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.988+0000 7ffa0c013800 20 remove_watcher() i=5 2024-05-01T13:02:06.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.988+0000 7ffa0c013800 20 remove_watcher() i=6 2024-05-01T13:02:06.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:06.990+0000 7ffa0c013800 20 remove_watcher() i=7 2024-05-01T13:02:12.049 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:12.049 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.087+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.089+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.089+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.090+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.091+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.091+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.092+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.092+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.092+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.092+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 zone MyZone found 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.093+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.094+0000 7f565652f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.094+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.094+0000 7f565652f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.094+0000 7f565652f800 20 started sync module instance, tier type = 2024-05-01T13:02:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.094+0000 7f565652f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:12.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=0 2024-05-01T13:02:12.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=1 2024-05-01T13:02:12.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=2 2024-05-01T13:02:12.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=3 2024-05-01T13:02:12.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=4 2024-05-01T13:02:12.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=5 2024-05-01T13:02:12.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.105+0000 7f565652f800 20 add_watcher() i=6 2024-05-01T13:02:12.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.106+0000 7f565652f800 20 add_watcher() i=7 2024-05-01T13:02:12.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.106+0000 7f565652f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:12.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.108+0000 7f5508fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f565652f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f565652f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f565652f800 5 note: GC not initialized 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f54837fe640 20 reqs_thread_entry: start 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f565652f800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f565652f800 20 Filter name: none 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.109+0000 7f54817fa640 20 reqs_thread_entry: start 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.110+0000 7f565652f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:12.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.110+0000 7f565652f800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:12.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.111+0000 7f5508fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.114+0000 7f565652f800 20 remove_watcher() i=0 2024-05-01T13:02:12.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.114+0000 7f565652f800 2 removed watcher, disabling cache 2024-05-01T13:02:12.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.115+0000 7f565652f800 20 remove_watcher() i=1 2024-05-01T13:02:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.116+0000 7f565652f800 20 remove_watcher() i=2 2024-05-01T13:02:12.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.117+0000 7f565652f800 20 remove_watcher() i=3 2024-05-01T13:02:12.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.118+0000 7f565652f800 20 remove_watcher() i=4 2024-05-01T13:02:12.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.119+0000 7f565652f800 20 remove_watcher() i=5 2024-05-01T13:02:12.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.120+0000 7f565652f800 20 remove_watcher() i=6 2024-05-01T13:02:12.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:12.122+0000 7f565652f800 20 remove_watcher() i=7 2024-05-01T13:02:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:17.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.218+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.220+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.220+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.221+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.221+0000 7f2200a3d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.221+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.221+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.221+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.222+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.222+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.223+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.224+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.224+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 zone MyZone found 2024-05-01T13:02:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:17.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 started sync module instance, tier type = 2024-05-01T13:02:17.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.225+0000 7f2200a3d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=0 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=1 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=2 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=3 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=4 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.237+0000 7f2200a3d800 20 add_watcher() i=5 2024-05-01T13:02:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.238+0000 7f2200a3d800 20 add_watcher() i=6 2024-05-01T13:02:17.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.239+0000 7f2200a3d800 20 add_watcher() i=7 2024-05-01T13:02:17.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.239+0000 7f2200a3d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.240+0000 7f20b4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f2200a3d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f2200a3d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f2200a3d800 5 note: GC not initialized 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f202f7fe640 20 reqs_thread_entry: start 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f2200a3d800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f2200a3d800 20 Filter name: none 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.241+0000 7f202d7fa640 20 reqs_thread_entry: start 2024-05-01T13:02:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.242+0000 7f2200a3d800 20 rados->read ofs=0 len=0 2024-05-01T13:02:17.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.242+0000 7f2200a3d800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:17.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.244+0000 7f20b4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.246+0000 7f2200a3d800 20 remove_watcher() i=0 2024-05-01T13:02:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.246+0000 7f2200a3d800 2 removed watcher, disabling cache 2024-05-01T13:02:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.247+0000 7f2200a3d800 20 remove_watcher() i=1 2024-05-01T13:02:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.248+0000 7f2200a3d800 20 remove_watcher() i=2 2024-05-01T13:02:17.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.249+0000 7f2200a3d800 20 remove_watcher() i=3 2024-05-01T13:02:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.250+0000 7f2200a3d800 20 remove_watcher() i=4 2024-05-01T13:02:17.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.251+0000 7f2200a3d800 20 remove_watcher() i=5 2024-05-01T13:02:17.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.252+0000 7f2200a3d800 20 remove_watcher() i=6 2024-05-01T13:02:17.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:17.253+0000 7f2200a3d800 20 remove_watcher() i=7 2024-05-01T13:02:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:22.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.350+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.352+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.352+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.352+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.352+0000 7f4c5c60e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.352+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.353+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.353+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.353+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.353+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.354+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.354+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.355+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.355+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.355+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.355+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 zone MyZone found 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.356+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.357+0000 7f4c5c60e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.357+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.357+0000 7f4c5c60e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.357+0000 7f4c5c60e800 20 started sync module instance, tier type = 2024-05-01T13:02:22.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.357+0000 7f4c5c60e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=0 2024-05-01T13:02:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=1 2024-05-01T13:02:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=2 2024-05-01T13:02:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=3 2024-05-01T13:02:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=4 2024-05-01T13:02:22.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=5 2024-05-01T13:02:22.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.369+0000 7f4c5c60e800 20 add_watcher() i=6 2024-05-01T13:02:22.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.370+0000 7f4c5c60e800 20 add_watcher() i=7 2024-05-01T13:02:22.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.370+0000 7f4c5c60e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4b10fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4c5c60e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4c5c60e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4c5c60e800 5 note: GC not initialized 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4a8b7fe640 20 reqs_thread_entry: start 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4c5c60e800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4c5c60e800 20 Filter name: none 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.372+0000 7f4a897fa640 20 reqs_thread_entry: start 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.373+0000 7f4c5c60e800 20 rados->read ofs=0 len=0 2024-05-01T13:02:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.373+0000 7f4c5c60e800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:22.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.375+0000 7f4b10fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:22.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.377+0000 7f4c5c60e800 20 remove_watcher() i=0 2024-05-01T13:02:22.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.377+0000 7f4c5c60e800 2 removed watcher, disabling cache 2024-05-01T13:02:22.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.378+0000 7f4c5c60e800 20 remove_watcher() i=1 2024-05-01T13:02:22.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.379+0000 7f4c5c60e800 20 remove_watcher() i=2 2024-05-01T13:02:22.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.380+0000 7f4c5c60e800 20 remove_watcher() i=3 2024-05-01T13:02:22.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.381+0000 7f4c5c60e800 20 remove_watcher() i=4 2024-05-01T13:02:22.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.382+0000 7f4c5c60e800 20 remove_watcher() i=5 2024-05-01T13:02:22.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.383+0000 7f4c5c60e800 20 remove_watcher() i=6 2024-05-01T13:02:22.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:22.385+0000 7f4c5c60e800 20 remove_watcher() i=7 2024-05-01T13:02:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.482+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.484+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.484+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.485+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.485+0000 7f3331d33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.485+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.486+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.486+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.486+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.486+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.487+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.487+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.487+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.487+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.488+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.488+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.488+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.488+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 zone MyZone found 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.489+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.490+0000 7f3331d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.490+0000 7f3331d33800 20 started sync module instance, tier type = 2024-05-01T13:02:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.490+0000 7f3331d33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=0 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=1 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=2 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=3 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=4 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=5 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.502+0000 7f3331d33800 20 add_watcher() i=6 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.503+0000 7f3331d33800 20 add_watcher() i=7 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.503+0000 7f3331d33800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f31df7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f3331d33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f3331d33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f3331d33800 5 note: GC not initialized 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f31de7fc640 20 reqs_thread_entry: start 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f3331d33800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f3331d33800 20 Filter name: none 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.505+0000 7f314ffff640 20 reqs_thread_entry: start 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.506+0000 7f3331d33800 20 rados->read ofs=0 len=0 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.506+0000 7f3331d33800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.507+0000 7f31df7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.510+0000 7f3331d33800 20 remove_watcher() i=0 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.510+0000 7f3331d33800 2 removed watcher, disabling cache 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.511+0000 7f3331d33800 20 remove_watcher() i=1 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.512+0000 7f3331d33800 20 remove_watcher() i=2 2024-05-01T13:02:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.513+0000 7f3331d33800 20 remove_watcher() i=3 2024-05-01T13:02:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.514+0000 7f3331d33800 20 remove_watcher() i=4 2024-05-01T13:02:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.515+0000 7f3331d33800 20 remove_watcher() i=5 2024-05-01T13:02:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.516+0000 7f3331d33800 20 remove_watcher() i=6 2024-05-01T13:02:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:27.517+0000 7f3331d33800 20 remove_watcher() i=7 2024-05-01T13:02:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.616+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.618+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.618+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.619+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.619+0000 7faff6933800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.619+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.619+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.619+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.620+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:32.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.620+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:32.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.621+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.622+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.622+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:32.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 zone MyZone found 2024-05-01T13:02:32.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:32.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 started sync module instance, tier type = 2024-05-01T13:02:32.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.623+0000 7faff6933800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=0 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=1 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=2 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=3 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=4 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=5 2024-05-01T13:02:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.634+0000 7faff6933800 20 add_watcher() i=6 2024-05-01T13:02:32.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.636+0000 7faff6933800 20 add_watcher() i=7 2024-05-01T13:02:32.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.636+0000 7faff6933800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faea8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faff6933800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faff6933800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faff6933800 5 note: GC not initialized 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7fae237fe640 20 reqs_thread_entry: start 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faff6933800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7faff6933800 20 Filter name: none 2024-05-01T13:02:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.638+0000 7fae217fa640 20 reqs_thread_entry: start 2024-05-01T13:02:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.639+0000 7faff6933800 20 rados->read ofs=0 len=0 2024-05-01T13:02:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.639+0000 7faff6933800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:32.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.641+0000 7faea8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:32.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.643+0000 7faff6933800 20 remove_watcher() i=0 2024-05-01T13:02:32.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.643+0000 7faff6933800 2 removed watcher, disabling cache 2024-05-01T13:02:32.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.644+0000 7faff6933800 20 remove_watcher() i=1 2024-05-01T13:02:32.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.646+0000 7faff6933800 20 remove_watcher() i=2 2024-05-01T13:02:32.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.647+0000 7faff6933800 20 remove_watcher() i=3 2024-05-01T13:02:32.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.648+0000 7faff6933800 20 remove_watcher() i=4 2024-05-01T13:02:32.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.649+0000 7faff6933800 20 remove_watcher() i=5 2024-05-01T13:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.650+0000 7faff6933800 20 remove_watcher() i=6 2024-05-01T13:02:32.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:32.652+0000 7faff6933800 20 remove_watcher() i=7 2024-05-01T13:02:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.748+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.750+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.750+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.751+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.751+0000 7efd5cf2f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.751+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.751+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.751+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.752+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.752+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.753+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.753+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.753+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.753+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.754+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.754+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.754+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.754+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 zone MyZone found 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.755+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.756+0000 7efd5cf2f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.756+0000 7efd5cf2f800 20 started sync module instance, tier type = 2024-05-01T13:02:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.756+0000 7efd5cf2f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=0 2024-05-01T13:02:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=1 2024-05-01T13:02:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=2 2024-05-01T13:02:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=3 2024-05-01T13:02:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=4 2024-05-01T13:02:37.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=5 2024-05-01T13:02:37.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.766+0000 7efd5cf2f800 20 add_watcher() i=6 2024-05-01T13:02:37.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.767+0000 7efd5cf2f800 20 add_watcher() i=7 2024-05-01T13:02:37.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.767+0000 7efd5cf2f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:37.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.769+0000 7efc0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:37.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efd5cf2f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:37.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efd5cf2f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efd5cf2f800 5 note: GC not initialized 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efc0dfe3640 20 reqs_thread_entry: start 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efd5cf2f800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efd5cf2f800 20 Filter name: none 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.770+0000 7efb7f7fe640 20 reqs_thread_entry: start 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.771+0000 7efd5cf2f800 20 rados->read ofs=0 len=0 2024-05-01T13:02:37.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.771+0000 7efd5cf2f800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:37.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.773+0000 7efc0efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:37.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.775+0000 7efd5cf2f800 20 remove_watcher() i=0 2024-05-01T13:02:37.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.775+0000 7efd5cf2f800 2 removed watcher, disabling cache 2024-05-01T13:02:37.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.776+0000 7efd5cf2f800 20 remove_watcher() i=1 2024-05-01T13:02:37.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.777+0000 7efd5cf2f800 20 remove_watcher() i=2 2024-05-01T13:02:37.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.778+0000 7efd5cf2f800 20 remove_watcher() i=3 2024-05-01T13:02:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.779+0000 7efd5cf2f800 20 remove_watcher() i=4 2024-05-01T13:02:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.780+0000 7efd5cf2f800 20 remove_watcher() i=5 2024-05-01T13:02:37.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.781+0000 7efd5cf2f800 20 remove_watcher() i=6 2024-05-01T13:02:37.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:37.782+0000 7efd5cf2f800 20 remove_watcher() i=7 2024-05-01T13:02:42.842 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:42.842 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:42.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.879+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.881+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:42.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.881+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.882+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:42.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.882+0000 7f9c83123800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:42.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.882+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.882+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:42.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.882+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.883+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:42.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.883+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:42.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.884+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.885+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:42.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.885+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 zone MyZone found 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:42.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 started sync module instance, tier type = 2024-05-01T13:02:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.886+0000 7f9c83123800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=0 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=1 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=2 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=3 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=4 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=5 2024-05-01T13:02:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.897+0000 7f9c83123800 20 add_watcher() i=6 2024-05-01T13:02:42.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.900+0000 7f9c83123800 20 add_watcher() i=7 2024-05-01T13:02:42.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.900+0000 7f9c83123800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:42.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.902+0000 7f9b357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:42.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.902+0000 7f9c83123800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:42.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.902+0000 7f9c83123800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:42.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.902+0000 7f9c83123800 5 note: GC not initialized 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.903+0000 7f9aaffff640 20 reqs_thread_entry: start 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.903+0000 7f9c83123800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.903+0000 7f9c83123800 20 Filter name: none 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.903+0000 7f9aadffb640 20 reqs_thread_entry: start 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.903+0000 7f9c83123800 20 rados->read ofs=0 len=0 2024-05-01T13:02:42.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.904+0000 7f9c83123800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:42.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.905+0000 7f9b357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:42.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.908+0000 7f9c83123800 20 remove_watcher() i=0 2024-05-01T13:02:42.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.908+0000 7f9c83123800 2 removed watcher, disabling cache 2024-05-01T13:02:42.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.909+0000 7f9c83123800 20 remove_watcher() i=1 2024-05-01T13:02:42.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.910+0000 7f9c83123800 20 remove_watcher() i=2 2024-05-01T13:02:42.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.911+0000 7f9c83123800 20 remove_watcher() i=3 2024-05-01T13:02:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.912+0000 7f9c83123800 20 remove_watcher() i=4 2024-05-01T13:02:42.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.913+0000 7f9c83123800 20 remove_watcher() i=5 2024-05-01T13:02:42.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.914+0000 7f9c83123800 20 remove_watcher() i=6 2024-05-01T13:02:42.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:42.915+0000 7f9c83123800 20 remove_watcher() i=7 2024-05-01T13:02:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.008+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.010+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:48.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.010+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.011+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.011+0000 7f5ab9e13800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.011+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.012+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.012+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.012+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:48.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.012+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.013+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.013+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.013+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.013+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.014+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.014+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.014+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.014+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 zone MyZone found 2024-05-01T13:02:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.015+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.016+0000 7f5ab9e13800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.016+0000 7f5ab9e13800 20 started sync module instance, tier type = 2024-05-01T13:02:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.016+0000 7f5ab9e13800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:48.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=0 2024-05-01T13:02:48.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=1 2024-05-01T13:02:48.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=2 2024-05-01T13:02:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=3 2024-05-01T13:02:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=4 2024-05-01T13:02:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=5 2024-05-01T13:02:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.028+0000 7f5ab9e13800 20 add_watcher() i=6 2024-05-01T13:02:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.029+0000 7f5ab9e13800 20 add_watcher() i=7 2024-05-01T13:02:48.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.029+0000 7f5ab9e13800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.031+0000 7f596b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.031+0000 7f5ab9e13800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.031+0000 7f5ab9e13800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.031+0000 7f5ab9e13800 5 note: GC not initialized 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.031+0000 7f596a7e4640 20 reqs_thread_entry: start 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.032+0000 7f5ab9e13800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:48.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.032+0000 7f5ab9e13800 20 Filter name: none 2024-05-01T13:02:48.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.032+0000 7f58dbfff640 20 reqs_thread_entry: start 2024-05-01T13:02:48.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.032+0000 7f5ab9e13800 20 rados->read ofs=0 len=0 2024-05-01T13:02:48.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.033+0000 7f5ab9e13800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:48.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.034+0000 7f596b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.037+0000 7f5ab9e13800 20 remove_watcher() i=0 2024-05-01T13:02:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.037+0000 7f5ab9e13800 2 removed watcher, disabling cache 2024-05-01T13:02:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.038+0000 7f5ab9e13800 20 remove_watcher() i=1 2024-05-01T13:02:48.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.039+0000 7f5ab9e13800 20 remove_watcher() i=2 2024-05-01T13:02:48.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.040+0000 7f5ab9e13800 20 remove_watcher() i=3 2024-05-01T13:02:48.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.041+0000 7f5ab9e13800 20 remove_watcher() i=4 2024-05-01T13:02:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.042+0000 7f5ab9e13800 20 remove_watcher() i=5 2024-05-01T13:02:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.043+0000 7f5ab9e13800 20 remove_watcher() i=6 2024-05-01T13:02:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:48.044+0000 7f5ab9e13800 20 remove_watcher() i=7 2024-05-01T13:02:53.103 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:53.103 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:53.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.141+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.142+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:53.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.142+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.143+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:53.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.143+0000 7f4bdda62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:53.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.143+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.143+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:53.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.143+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.144+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.144+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:53.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.145+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.146+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:53.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.146+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 zone MyZone found 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.147+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.148+0000 7f4bdda62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.148+0000 7f4bdda62800 20 started sync module instance, tier type = 2024-05-01T13:02:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.148+0000 7f4bdda62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=0 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=1 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=2 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=3 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=4 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=5 2024-05-01T13:02:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.158+0000 7f4bdda62800 20 add_watcher() i=6 2024-05-01T13:02:53.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.160+0000 7f4bdda62800 20 add_watcher() i=7 2024-05-01T13:02:53.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.160+0000 7f4bdda62800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.161+0000 7f4a91fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4bdda62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4bdda62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4bdda62800 5 note: GC not initialized 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4a90fe1640 20 reqs_thread_entry: start 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4bdda62800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4bdda62800 20 Filter name: none 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.162+0000 7f4a0a7fc640 20 reqs_thread_entry: start 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.163+0000 7f4bdda62800 20 rados->read ofs=0 len=0 2024-05-01T13:02:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.163+0000 7f4bdda62800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:53.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.165+0000 7f4a91fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.168+0000 7f4bdda62800 20 remove_watcher() i=0 2024-05-01T13:02:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.168+0000 7f4bdda62800 2 removed watcher, disabling cache 2024-05-01T13:02:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.169+0000 7f4bdda62800 20 remove_watcher() i=1 2024-05-01T13:02:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.170+0000 7f4bdda62800 20 remove_watcher() i=2 2024-05-01T13:02:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.171+0000 7f4bdda62800 20 remove_watcher() i=3 2024-05-01T13:02:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.172+0000 7f4bdda62800 20 remove_watcher() i=4 2024-05-01T13:02:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.173+0000 7f4bdda62800 20 remove_watcher() i=5 2024-05-01T13:02:53.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.174+0000 7f4bdda62800 20 remove_watcher() i=6 2024-05-01T13:02:53.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:53.175+0000 7f4bdda62800 20 remove_watcher() i=7 2024-05-01T13:02:58.233 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:02:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.270+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.272+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.272+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.273+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.273+0000 7fda07542800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.273+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.273+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.273+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.274+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.274+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.275+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.276+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.276+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.276+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 zone MyZone found 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:02:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 period zonegroup name MyZoneGroup 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 started sync module instance, tier type = 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.277+0000 7fda07542800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=0 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=1 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=2 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=3 2024-05-01T13:02:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=4 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=5 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.288+0000 7fda07542800 20 add_watcher() i=6 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.289+0000 7fda07542800 20 add_watcher() i=7 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.289+0000 7fda07542800 2 all 8 watchers are set, enabling cache 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fd8bb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fda07542800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fda07542800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fda07542800 5 note: GC not initialized 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fd8ba7e4640 20 reqs_thread_entry: start 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fda07542800 20 init_complete bucket index max shards: 11 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fda07542800 20 Filter name: none 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.291+0000 7fd82bfff640 20 reqs_thread_entry: start 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.292+0000 7fda07542800 20 rados->read ofs=0 len=0 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.292+0000 7fda07542800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:02:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.294+0000 7fd8bb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.296+0000 7fda07542800 20 remove_watcher() i=0 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.296+0000 7fda07542800 2 removed watcher, disabling cache 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.297+0000 7fda07542800 20 remove_watcher() i=1 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.298+0000 7fda07542800 20 remove_watcher() i=2 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.299+0000 7fda07542800 20 remove_watcher() i=3 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.300+0000 7fda07542800 20 remove_watcher() i=4 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.301+0000 7fda07542800 20 remove_watcher() i=5 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.303+0000 7fda07542800 20 remove_watcher() i=6 2024-05-01T13:02:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:02:58.304+0000 7fda07542800 20 remove_watcher() i=7 2024-05-01T13:03:03.362 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:03.362 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.398+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.400+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.400+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.401+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.401+0000 7f3866d25800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.401+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.401+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.401+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.402+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.402+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.403+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 zone MyZone found 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.404+0000 7f3866d25800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 started sync module instance, tier type = 2024-05-01T13:03:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.405+0000 7f3866d25800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=0 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=1 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=2 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=3 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=4 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.418+0000 7f3866d25800 20 add_watcher() i=5 2024-05-01T13:03:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.419+0000 7f3866d25800 20 add_watcher() i=6 2024-05-01T13:03:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.420+0000 7f3866d25800 20 add_watcher() i=7 2024-05-01T13:03:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.420+0000 7f3866d25800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:03.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.421+0000 7f37197e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3866d25800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3866d25800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3866d25800 5 note: GC not initialized 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3693fff640 20 reqs_thread_entry: start 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3866d25800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3866d25800 20 Filter name: none 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.422+0000 7f3691ffb640 20 reqs_thread_entry: start 2024-05-01T13:03:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.423+0000 7f3866d25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:03.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.423+0000 7f3866d25800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.425+0000 7f37197e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.427+0000 7f3866d25800 20 remove_watcher() i=0 2024-05-01T13:03:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.427+0000 7f3866d25800 2 removed watcher, disabling cache 2024-05-01T13:03:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.428+0000 7f3866d25800 20 remove_watcher() i=1 2024-05-01T13:03:03.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.429+0000 7f3866d25800 20 remove_watcher() i=2 2024-05-01T13:03:03.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.430+0000 7f3866d25800 20 remove_watcher() i=3 2024-05-01T13:03:03.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.431+0000 7f3866d25800 20 remove_watcher() i=4 2024-05-01T13:03:03.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.432+0000 7f3866d25800 20 remove_watcher() i=5 2024-05-01T13:03:03.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.433+0000 7f3866d25800 20 remove_watcher() i=6 2024-05-01T13:03:03.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:03.435+0000 7f3866d25800 20 remove_watcher() i=7 2024-05-01T13:03:08.491 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:08.492 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:08.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.528+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.530+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:08.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.530+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.531+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.531+0000 7fa880125800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.531+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.532+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.532+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.532+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.532+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.533+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.533+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.533+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.533+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.534+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.534+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.534+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:08.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.534+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 20 zone MyZone found 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.535+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.536+0000 7fa880125800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.536+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.536+0000 7fa880125800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.536+0000 7fa880125800 20 started sync module instance, tier type = 2024-05-01T13:03:08.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.536+0000 7fa880125800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=0 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=1 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=2 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=3 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=4 2024-05-01T13:03:08.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=5 2024-05-01T13:03:08.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.547+0000 7fa880125800 20 add_watcher() i=6 2024-05-01T13:03:08.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.548+0000 7fa880125800 20 add_watcher() i=7 2024-05-01T13:03:08.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.548+0000 7fa880125800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa731fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa880125800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa880125800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa880125800 5 note: GC not initialized 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa730fe1640 20 reqs_thread_entry: start 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa880125800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.550+0000 7fa880125800 20 Filter name: none 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.551+0000 7fa6aa7fc640 20 reqs_thread_entry: start 2024-05-01T13:03:08.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.551+0000 7fa880125800 20 rados->read ofs=0 len=0 2024-05-01T13:03:08.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.551+0000 7fa880125800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:08.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.553+0000 7fa731fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.555+0000 7fa880125800 20 remove_watcher() i=0 2024-05-01T13:03:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.555+0000 7fa880125800 2 removed watcher, disabling cache 2024-05-01T13:03:08.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.556+0000 7fa880125800 20 remove_watcher() i=1 2024-05-01T13:03:08.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.557+0000 7fa880125800 20 remove_watcher() i=2 2024-05-01T13:03:08.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.559+0000 7fa880125800 20 remove_watcher() i=3 2024-05-01T13:03:08.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.560+0000 7fa880125800 20 remove_watcher() i=4 2024-05-01T13:03:08.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.561+0000 7fa880125800 20 remove_watcher() i=5 2024-05-01T13:03:08.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.562+0000 7fa880125800 20 remove_watcher() i=6 2024-05-01T13:03:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:08.563+0000 7fa880125800 20 remove_watcher() i=7 2024-05-01T13:03:13.621 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:13.621 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.659+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.661+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.661+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.662+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.662+0000 7fe2e913f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.662+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.662+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.662+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.663+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.663+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.664+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.665+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.665+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 zone MyZone found 2024-05-01T13:03:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 started sync module instance, tier type = 2024-05-01T13:03:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.666+0000 7fe2e913f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=0 2024-05-01T13:03:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=1 2024-05-01T13:03:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=2 2024-05-01T13:03:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=3 2024-05-01T13:03:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=4 2024-05-01T13:03:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=5 2024-05-01T13:03:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.678+0000 7fe2e913f800 20 add_watcher() i=6 2024-05-01T13:03:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.679+0000 7fe2e913f800 20 add_watcher() i=7 2024-05-01T13:03:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.679+0000 7fe2e913f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:13.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.681+0000 7fe199fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe2e913f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe2e913f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe2e913f800 5 note: GC not initialized 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe198fe1640 20 reqs_thread_entry: start 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe2e913f800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe2e913f800 20 Filter name: none 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.682+0000 7fe1127fc640 20 reqs_thread_entry: start 2024-05-01T13:03:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.683+0000 7fe2e913f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.683+0000 7fe2e913f800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:13.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.685+0000 7fe199fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.687+0000 7fe2e913f800 20 remove_watcher() i=0 2024-05-01T13:03:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.687+0000 7fe2e913f800 2 removed watcher, disabling cache 2024-05-01T13:03:13.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.688+0000 7fe2e913f800 20 remove_watcher() i=1 2024-05-01T13:03:13.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.689+0000 7fe2e913f800 20 remove_watcher() i=2 2024-05-01T13:03:13.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.690+0000 7fe2e913f800 20 remove_watcher() i=3 2024-05-01T13:03:13.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.691+0000 7fe2e913f800 20 remove_watcher() i=4 2024-05-01T13:03:13.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.693+0000 7fe2e913f800 20 remove_watcher() i=5 2024-05-01T13:03:13.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.694+0000 7fe2e913f800 20 remove_watcher() i=6 2024-05-01T13:03:13.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:13.695+0000 7fe2e913f800 20 remove_watcher() i=7 2024-05-01T13:03:18.753 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:18.754 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:18.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.791+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.792+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:18.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.792+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.793+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.793+0000 7faf1f91f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.793+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.793+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.793+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.794+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:18.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.794+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.795+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:18.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.795+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.795+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:18.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.795+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.796+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:18.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.796+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.796+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:18.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.796+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 zone MyZone found 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.797+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.798+0000 7faf1f91f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.798+0000 7faf1f91f800 20 started sync module instance, tier type = 2024-05-01T13:03:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.798+0000 7faf1f91f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:18.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=0 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=1 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=2 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=3 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=4 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=5 2024-05-01T13:03:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.809+0000 7faf1f91f800 20 add_watcher() i=6 2024-05-01T13:03:18.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.810+0000 7faf1f91f800 20 add_watcher() i=7 2024-05-01T13:03:18.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.810+0000 7faf1f91f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:18.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.812+0000 7fadd1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:18.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:18.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:18.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 5 note: GC not initialized 2024-05-01T13:03:18.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7fadd0fe1640 20 reqs_thread_entry: start 2024-05-01T13:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 20 Filter name: none 2024-05-01T13:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7fad4a7fc640 20 reqs_thread_entry: start 2024-05-01T13:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.813+0000 7faf1f91f800 20 rados->read ofs=0 len=0 2024-05-01T13:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.814+0000 7faf1f91f800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:18.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.816+0000 7fadd1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:18.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.818+0000 7faf1f91f800 20 remove_watcher() i=0 2024-05-01T13:03:18.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.818+0000 7faf1f91f800 2 removed watcher, disabling cache 2024-05-01T13:03:18.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.819+0000 7faf1f91f800 20 remove_watcher() i=1 2024-05-01T13:03:18.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.820+0000 7faf1f91f800 20 remove_watcher() i=2 2024-05-01T13:03:18.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.821+0000 7faf1f91f800 20 remove_watcher() i=3 2024-05-01T13:03:18.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.822+0000 7faf1f91f800 20 remove_watcher() i=4 2024-05-01T13:03:18.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.823+0000 7faf1f91f800 20 remove_watcher() i=5 2024-05-01T13:03:18.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.825+0000 7faf1f91f800 20 remove_watcher() i=6 2024-05-01T13:03:18.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:18.826+0000 7faf1f91f800 20 remove_watcher() i=7 2024-05-01T13:03:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:23.885 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:23.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.922+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.924+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:23.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.924+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.925+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.925+0000 7f8b2d830800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.925+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.925+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.925+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.926+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.926+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.926+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:23.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.926+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.927+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:23.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.927+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.927+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:23.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.927+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.927+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:23.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:23.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 zone MyZone found 2024-05-01T13:03:23.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:23.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.928+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.929+0000 7f8b2d830800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.929+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.929+0000 7f8b2d830800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.929+0000 7f8b2d830800 20 started sync module instance, tier type = 2024-05-01T13:03:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.929+0000 7f8b2d830800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=0 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=1 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=2 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=3 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=4 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=5 2024-05-01T13:03:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.940+0000 7f8b2d830800 20 add_watcher() i=6 2024-05-01T13:03:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.941+0000 7f8b2d830800 20 add_watcher() i=7 2024-05-01T13:03:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.941+0000 7f8b2d830800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f89defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f8b2d830800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f8b2d830800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f8b2d830800 5 note: GC not initialized 2024-05-01T13:03:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f89ddfe3640 20 reqs_thread_entry: start 2024-05-01T13:03:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.943+0000 7f8b2d830800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.944+0000 7f8b2d830800 20 Filter name: none 2024-05-01T13:03:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.944+0000 7f894f7fe640 20 reqs_thread_entry: start 2024-05-01T13:03:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.944+0000 7f8b2d830800 20 rados->read ofs=0 len=0 2024-05-01T13:03:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.944+0000 7f8b2d830800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:23.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.946+0000 7f89defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:23.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.948+0000 7f8b2d830800 20 remove_watcher() i=0 2024-05-01T13:03:23.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.948+0000 7f8b2d830800 2 removed watcher, disabling cache 2024-05-01T13:03:23.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.950+0000 7f8b2d830800 20 remove_watcher() i=1 2024-05-01T13:03:23.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.951+0000 7f8b2d830800 20 remove_watcher() i=2 2024-05-01T13:03:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.952+0000 7f8b2d830800 20 remove_watcher() i=3 2024-05-01T13:03:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.953+0000 7f8b2d830800 20 remove_watcher() i=4 2024-05-01T13:03:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.954+0000 7f8b2d830800 20 remove_watcher() i=5 2024-05-01T13:03:23.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.955+0000 7f8b2d830800 20 remove_watcher() i=6 2024-05-01T13:03:23.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:23.956+0000 7f8b2d830800 20 remove_watcher() i=7 2024-05-01T13:03:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:29.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.052+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.054+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:29.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.054+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.055+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.055+0000 7f593ff25800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.055+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.055+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.055+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.056+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:29.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.056+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.056+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.057+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.057+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.057+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.057+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:29.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.057+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:29.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 20 zone MyZone found 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.058+0000 7f593ff25800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 started sync module instance, tier type = 2024-05-01T13:03:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.059+0000 7f593ff25800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=0 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=1 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=2 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=3 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=4 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=5 2024-05-01T13:03:29.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.072+0000 7f593ff25800 20 add_watcher() i=6 2024-05-01T13:03:29.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.073+0000 7f593ff25800 20 add_watcher() i=7 2024-05-01T13:03:29.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.073+0000 7f593ff25800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:29.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.075+0000 7f57f1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:29.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:29.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 5 note: GC not initialized 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f57f0fe1640 20 reqs_thread_entry: start 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 20 Filter name: none 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f576a7fc640 20 reqs_thread_entry: start 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.076+0000 7f593ff25800 20 rados->read ofs=0 len=0 2024-05-01T13:03:29.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.077+0000 7f593ff25800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:29.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.079+0000 7f57f1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:29.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.081+0000 7f593ff25800 20 remove_watcher() i=0 2024-05-01T13:03:29.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.081+0000 7f593ff25800 2 removed watcher, disabling cache 2024-05-01T13:03:29.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.082+0000 7f593ff25800 20 remove_watcher() i=1 2024-05-01T13:03:29.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.083+0000 7f593ff25800 20 remove_watcher() i=2 2024-05-01T13:03:29.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.084+0000 7f593ff25800 20 remove_watcher() i=3 2024-05-01T13:03:29.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.085+0000 7f593ff25800 20 remove_watcher() i=4 2024-05-01T13:03:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.087+0000 7f593ff25800 20 remove_watcher() i=5 2024-05-01T13:03:29.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.088+0000 7f593ff25800 20 remove_watcher() i=6 2024-05-01T13:03:29.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:29.089+0000 7f593ff25800 20 remove_watcher() i=7 2024-05-01T13:03:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.188+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.190+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:34.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.190+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.191+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:34.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.191+0000 7efe6880e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:34.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.191+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.191+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:34.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.191+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.192+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:34.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.192+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.193+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:34.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.193+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.193+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:34.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.193+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.194+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:34.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.194+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.194+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:34.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.194+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 zone MyZone found 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.195+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.196+0000 7efe6880e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.196+0000 7efe6880e800 20 started sync module instance, tier type = 2024-05-01T13:03:34.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.196+0000 7efe6880e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=0 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=1 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=2 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=3 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=4 2024-05-01T13:03:34.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=5 2024-05-01T13:03:34.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.207+0000 7efe6880e800 20 add_watcher() i=6 2024-05-01T13:03:34.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.208+0000 7efe6880e800 20 add_watcher() i=7 2024-05-01T13:03:34.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.208+0000 7efe6880e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:34.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.209+0000 7efd1cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efe6880e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efe6880e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efe6880e800 5 note: GC not initialized 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efc977fe640 20 reqs_thread_entry: start 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efe6880e800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efe6880e800 20 Filter name: none 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.210+0000 7efc957fa640 20 reqs_thread_entry: start 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.211+0000 7efe6880e800 20 rados->read ofs=0 len=0 2024-05-01T13:03:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.211+0000 7efe6880e800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:34.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.213+0000 7efd1cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:34.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.215+0000 7efe6880e800 20 remove_watcher() i=0 2024-05-01T13:03:34.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.215+0000 7efe6880e800 2 removed watcher, disabling cache 2024-05-01T13:03:34.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.216+0000 7efe6880e800 20 remove_watcher() i=1 2024-05-01T13:03:34.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.218+0000 7efe6880e800 20 remove_watcher() i=2 2024-05-01T13:03:34.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.219+0000 7efe6880e800 20 remove_watcher() i=3 2024-05-01T13:03:34.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.220+0000 7efe6880e800 20 remove_watcher() i=4 2024-05-01T13:03:34.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.221+0000 7efe6880e800 20 remove_watcher() i=5 2024-05-01T13:03:34.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.222+0000 7efe6880e800 20 remove_watcher() i=6 2024-05-01T13:03:34.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:34.223+0000 7efe6880e800 20 remove_watcher() i=7 2024-05-01T13:03:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.318+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.320+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.320+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.321+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.321+0000 7f7cf632d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.321+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.321+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.321+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.322+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.322+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.323+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.324+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.324+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.324+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 zone MyZone found 2024-05-01T13:03:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 started sync module instance, tier type = 2024-05-01T13:03:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.325+0000 7f7cf632d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=0 2024-05-01T13:03:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=1 2024-05-01T13:03:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=2 2024-05-01T13:03:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=3 2024-05-01T13:03:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=4 2024-05-01T13:03:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=5 2024-05-01T13:03:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.336+0000 7f7cf632d800 20 add_watcher() i=6 2024-05-01T13:03:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.337+0000 7f7cf632d800 20 add_watcher() i=7 2024-05-01T13:03:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.337+0000 7f7cf632d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.339+0000 7f7ba37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7cf632d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7cf632d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7cf632d800 5 note: GC not initialized 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7ba27fc640 20 reqs_thread_entry: start 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7cf632d800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7cf632d800 20 Filter name: none 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.340+0000 7f7b13fff640 20 reqs_thread_entry: start 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.341+0000 7f7cf632d800 20 rados->read ofs=0 len=0 2024-05-01T13:03:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.341+0000 7f7cf632d800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.343+0000 7f7ba37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.345+0000 7f7cf632d800 20 remove_watcher() i=0 2024-05-01T13:03:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.345+0000 7f7cf632d800 2 removed watcher, disabling cache 2024-05-01T13:03:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.346+0000 7f7cf632d800 20 remove_watcher() i=1 2024-05-01T13:03:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.347+0000 7f7cf632d800 20 remove_watcher() i=2 2024-05-01T13:03:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.348+0000 7f7cf632d800 20 remove_watcher() i=3 2024-05-01T13:03:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.349+0000 7f7cf632d800 20 remove_watcher() i=4 2024-05-01T13:03:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.350+0000 7f7cf632d800 20 remove_watcher() i=5 2024-05-01T13:03:39.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.351+0000 7f7cf632d800 20 remove_watcher() i=6 2024-05-01T13:03:39.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:39.352+0000 7f7cf632d800 20 remove_watcher() i=7 2024-05-01T13:03:44.410 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:44.410 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:44.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.447+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.449+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:44.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.449+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.449+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:44.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.450+0000 7fc55f098800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:44.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.450+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.450+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:44.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.450+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.451+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:44.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.451+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.451+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:44.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.451+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.452+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:44.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.452+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.452+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:44.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.452+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:44.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:44.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 zone MyZone found 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.453+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.454+0000 7fc55f098800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.454+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.454+0000 7fc55f098800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.454+0000 7fc55f098800 20 started sync module instance, tier type = 2024-05-01T13:03:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.454+0000 7fc55f098800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=0 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=1 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=2 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=3 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=4 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=5 2024-05-01T13:03:44.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.466+0000 7fc55f098800 20 add_watcher() i=6 2024-05-01T13:03:44.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.467+0000 7fc55f098800 20 add_watcher() i=7 2024-05-01T13:03:44.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.467+0000 7fc55f098800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc412fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc55f098800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc55f098800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc55f098800 5 note: GC not initialized 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc411fe3640 20 reqs_thread_entry: start 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc55f098800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.469+0000 7fc55f098800 20 Filter name: none 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.470+0000 7fc3837fe640 20 reqs_thread_entry: start 2024-05-01T13:03:44.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.470+0000 7fc55f098800 20 rados->read ofs=0 len=0 2024-05-01T13:03:44.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.470+0000 7fc55f098800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:44.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.472+0000 7fc412fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:44.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.475+0000 7fc55f098800 20 remove_watcher() i=0 2024-05-01T13:03:44.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.475+0000 7fc55f098800 2 removed watcher, disabling cache 2024-05-01T13:03:44.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.475+0000 7fc55f098800 20 remove_watcher() i=1 2024-05-01T13:03:44.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.476+0000 7fc55f098800 20 remove_watcher() i=2 2024-05-01T13:03:44.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.477+0000 7fc55f098800 20 remove_watcher() i=3 2024-05-01T13:03:44.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.479+0000 7fc55f098800 20 remove_watcher() i=4 2024-05-01T13:03:44.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.480+0000 7fc55f098800 20 remove_watcher() i=5 2024-05-01T13:03:44.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.481+0000 7fc55f098800 20 remove_watcher() i=6 2024-05-01T13:03:44.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:44.482+0000 7fc55f098800 20 remove_watcher() i=7 2024-05-01T13:03:49.542 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:49.542 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:49.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.581+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.583+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:49.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.583+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.583+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:49.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.583+0000 7f8c90f44800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:49.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.583+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.584+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:49.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.584+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.584+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:49.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.584+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.585+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:49.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.585+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.585+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:49.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.585+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.586+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:49.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.586+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:49.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 zone MyZone found 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.587+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.588+0000 7f8c90f44800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.588+0000 7f8c90f44800 20 started sync module instance, tier type = 2024-05-01T13:03:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.588+0000 7f8c90f44800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=0 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=1 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=2 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=3 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=4 2024-05-01T13:03:49.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=5 2024-05-01T13:03:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.599+0000 7f8c90f44800 20 add_watcher() i=6 2024-05-01T13:03:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.600+0000 7f8c90f44800 20 add_watcher() i=7 2024-05-01T13:03:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.600+0000 7f8c90f44800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.602+0000 7f8b457e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.602+0000 7f8c90f44800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.602+0000 7f8c90f44800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.602+0000 7f8c90f44800 5 note: GC not initialized 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.602+0000 7f8abffff640 20 reqs_thread_entry: start 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.603+0000 7f8c90f44800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.603+0000 7f8c90f44800 20 Filter name: none 2024-05-01T13:03:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.603+0000 7f8abdffb640 20 reqs_thread_entry: start 2024-05-01T13:03:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.603+0000 7f8c90f44800 20 rados->read ofs=0 len=0 2024-05-01T13:03:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.603+0000 7f8c90f44800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.605+0000 7f8b457e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.607+0000 7f8c90f44800 20 remove_watcher() i=0 2024-05-01T13:03:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.607+0000 7f8c90f44800 2 removed watcher, disabling cache 2024-05-01T13:03:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.608+0000 7f8c90f44800 20 remove_watcher() i=1 2024-05-01T13:03:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.609+0000 7f8c90f44800 20 remove_watcher() i=2 2024-05-01T13:03:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.610+0000 7f8c90f44800 20 remove_watcher() i=3 2024-05-01T13:03:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.611+0000 7f8c90f44800 20 remove_watcher() i=4 2024-05-01T13:03:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.612+0000 7f8c90f44800 20 remove_watcher() i=5 2024-05-01T13:03:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.613+0000 7f8c90f44800 20 remove_watcher() i=6 2024-05-01T13:03:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:49.614+0000 7f8c90f44800 20 remove_watcher() i=7 2024-05-01T13:03:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:54.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.711+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.712+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:54.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.712+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.713+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:54.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.713+0000 7fc195f27800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:54.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.713+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.713+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:54.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.713+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.714+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:54.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.714+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.715+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:54.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.715+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.715+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:54.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.715+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.716+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:54.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.716+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 zone MyZone found 2024-05-01T13:03:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.717+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.718+0000 7fc195f27800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.718+0000 7fc195f27800 20 started sync module instance, tier type = 2024-05-01T13:03:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.718+0000 7fc195f27800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=0 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=1 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=2 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=3 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=4 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=5 2024-05-01T13:03:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.729+0000 7fc195f27800 20 add_watcher() i=6 2024-05-01T13:03:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.730+0000 7fc195f27800 20 add_watcher() i=7 2024-05-01T13:03:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.730+0000 7fc195f27800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.731+0000 7fc0437fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc195f27800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc195f27800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc195f27800 5 note: GC not initialized 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc0427fc640 20 reqs_thread_entry: start 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc195f27800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fc195f27800 20 Filter name: none 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.732+0000 7fbfb3fff640 20 reqs_thread_entry: start 2024-05-01T13:03:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.733+0000 7fc195f27800 20 rados->read ofs=0 len=0 2024-05-01T13:03:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.733+0000 7fc195f27800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.734+0000 7fc0437fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.736+0000 7fc195f27800 20 remove_watcher() i=0 2024-05-01T13:03:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.736+0000 7fc195f27800 2 removed watcher, disabling cache 2024-05-01T13:03:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.737+0000 7fc195f27800 20 remove_watcher() i=1 2024-05-01T13:03:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.738+0000 7fc195f27800 20 remove_watcher() i=2 2024-05-01T13:03:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.740+0000 7fc195f27800 20 remove_watcher() i=3 2024-05-01T13:03:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.741+0000 7fc195f27800 20 remove_watcher() i=4 2024-05-01T13:03:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.742+0000 7fc195f27800 20 remove_watcher() i=5 2024-05-01T13:03:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.743+0000 7fc195f27800 20 remove_watcher() i=6 2024-05-01T13:03:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:54.744+0000 7fc195f27800 20 remove_watcher() i=7 2024-05-01T13:03:59.802 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:03:59.802 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:03:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.839+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.841+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.841+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.842+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.842+0000 7fb568813800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:03:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.842+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.842+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.842+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.843+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:03:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.843+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:03:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.844+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:03:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 zone MyZone found 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 period zonegroup name MyZoneGroup 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.845+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.846+0000 7fb568813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.846+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.846+0000 7fb568813800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.846+0000 7fb568813800 20 started sync module instance, tier type = 2024-05-01T13:03:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.846+0000 7fb568813800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=0 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=1 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=2 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=3 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=4 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=5 2024-05-01T13:03:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.857+0000 7fb568813800 20 add_watcher() i=6 2024-05-01T13:03:59.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.858+0000 7fb568813800 20 add_watcher() i=7 2024-05-01T13:03:59.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.858+0000 7fb568813800 2 all 8 watchers are set, enabling cache 2024-05-01T13:03:59.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.860+0000 7fb419fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.860+0000 7fb568813800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.860+0000 7fb568813800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.860+0000 7fb568813800 5 note: GC not initialized 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.861+0000 7fb418fe1640 20 reqs_thread_entry: start 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.861+0000 7fb568813800 20 init_complete bucket index max shards: 11 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.861+0000 7fb568813800 20 Filter name: none 2024-05-01T13:03:59.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.861+0000 7fb3927fc640 20 reqs_thread_entry: start 2024-05-01T13:03:59.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.861+0000 7fb568813800 20 rados->read ofs=0 len=0 2024-05-01T13:03:59.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.862+0000 7fb568813800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:03:59.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.863+0000 7fb419fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:03:59.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.865+0000 7fb568813800 20 remove_watcher() i=0 2024-05-01T13:03:59.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.865+0000 7fb568813800 2 removed watcher, disabling cache 2024-05-01T13:03:59.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.866+0000 7fb568813800 20 remove_watcher() i=1 2024-05-01T13:03:59.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.867+0000 7fb568813800 20 remove_watcher() i=2 2024-05-01T13:03:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.868+0000 7fb568813800 20 remove_watcher() i=3 2024-05-01T13:03:59.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.869+0000 7fb568813800 20 remove_watcher() i=4 2024-05-01T13:03:59.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.870+0000 7fb568813800 20 remove_watcher() i=5 2024-05-01T13:03:59.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.871+0000 7fb568813800 20 remove_watcher() i=6 2024-05-01T13:03:59.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:03:59.872+0000 7fb568813800 20 remove_watcher() i=7 2024-05-01T13:04:04.932 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:04.932 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.970+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.971+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.971+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.972+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.972+0000 7f838f210800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.972+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.972+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.972+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.973+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.973+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.974+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.974+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.974+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.974+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.975+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.975+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.975+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.975+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 20 zone MyZone found 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.976+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.977+0000 7f838f210800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.977+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.977+0000 7f838f210800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.977+0000 7f838f210800 20 started sync module instance, tier type = 2024-05-01T13:04:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.977+0000 7f838f210800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=0 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=1 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=2 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=3 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=4 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=5 2024-05-01T13:04:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.988+0000 7f838f210800 20 add_watcher() i=6 2024-05-01T13:04:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.989+0000 7f838f210800 20 add_watcher() i=7 2024-05-01T13:04:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.989+0000 7f838f210800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.991+0000 7f82417e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 5 note: GC not initialized 2024-05-01T13:04:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f81bbfff640 20 reqs_thread_entry: start 2024-05-01T13:04:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 20 Filter name: none 2024-05-01T13:04:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f81b9ffb640 20 reqs_thread_entry: start 2024-05-01T13:04:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.992+0000 7f838f210800 20 rados->read ofs=0 len=0 2024-05-01T13:04:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.993+0000 7f838f210800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:04:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.995+0000 7f82417e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.997+0000 7f838f210800 20 remove_watcher() i=0 2024-05-01T13:04:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.997+0000 7f838f210800 2 removed watcher, disabling cache 2024-05-01T13:04:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.998+0000 7f838f210800 20 remove_watcher() i=1 2024-05-01T13:04:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:04.999+0000 7f838f210800 20 remove_watcher() i=2 2024-05-01T13:04:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:05.000+0000 7f838f210800 20 remove_watcher() i=3 2024-05-01T13:04:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:05.001+0000 7f838f210800 20 remove_watcher() i=4 2024-05-01T13:04:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:05.002+0000 7f838f210800 20 remove_watcher() i=5 2024-05-01T13:04:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:05.004+0000 7f838f210800 20 remove_watcher() i=6 2024-05-01T13:04:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:05.005+0000 7f838f210800 20 remove_watcher() i=7 2024-05-01T13:04:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:10.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.100+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.102+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:10.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.102+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.103+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:10.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.103+0000 7fbf12f2d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:10.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.103+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.103+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:10.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.103+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.104+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:10.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.104+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.105+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:10.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.105+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.105+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.105+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.106+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.106+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.106+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.106+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 zone MyZone found 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.107+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.108+0000 7fbf12f2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.108+0000 7fbf12f2d800 20 started sync module instance, tier type = 2024-05-01T13:04:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.108+0000 7fbf12f2d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=0 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=1 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=2 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=3 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=4 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=5 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.118+0000 7fbf12f2d800 20 add_watcher() i=6 2024-05-01T13:04:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.119+0000 7fbf12f2d800 20 add_watcher() i=7 2024-05-01T13:04:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.119+0000 7fbf12f2d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.121+0000 7fbdc57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.121+0000 7fbf12f2d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.121+0000 7fbf12f2d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.121+0000 7fbf12f2d800 5 note: GC not initialized 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.122+0000 7fbd3ffff640 20 reqs_thread_entry: start 2024-05-01T13:04:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.122+0000 7fbf12f2d800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.122+0000 7fbf12f2d800 20 Filter name: none 2024-05-01T13:04:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.122+0000 7fbd3dffb640 20 reqs_thread_entry: start 2024-05-01T13:04:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.122+0000 7fbf12f2d800 20 rados->read ofs=0 len=0 2024-05-01T13:04:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.123+0000 7fbf12f2d800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:04:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.124+0000 7fbdc57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.127+0000 7fbf12f2d800 20 remove_watcher() i=0 2024-05-01T13:04:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.127+0000 7fbf12f2d800 2 removed watcher, disabling cache 2024-05-01T13:04:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.128+0000 7fbf12f2d800 20 remove_watcher() i=1 2024-05-01T13:04:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.129+0000 7fbf12f2d800 20 remove_watcher() i=2 2024-05-01T13:04:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.130+0000 7fbf12f2d800 20 remove_watcher() i=3 2024-05-01T13:04:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.131+0000 7fbf12f2d800 20 remove_watcher() i=4 2024-05-01T13:04:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.132+0000 7fbf12f2d800 20 remove_watcher() i=5 2024-05-01T13:04:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.134+0000 7fbf12f2d800 20 remove_watcher() i=6 2024-05-01T13:04:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:10.135+0000 7fbf12f2d800 20 remove_watcher() i=7 2024-05-01T13:04:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:FAIL 2024-05-01T13:04:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:test pushing persistent notification http ... 172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:11] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.718 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:12.718 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.757+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.759+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:12.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.759+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.760+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.760+0000 7f10df3a2800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.760+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.760+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.760+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.761+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.761+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:12.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:12.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:12.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.762+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.763+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:12.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.763+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 zone MyZone found 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:12.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 started sync module instance, tier type = 2024-05-01T13:04:12.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.764+0000 7f10df3a2800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=0 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=1 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=2 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=3 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=4 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=5 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.775+0000 7f10df3a2800 20 add_watcher() i=6 2024-05-01T13:04:12.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.776+0000 7f10df3a2800 20 add_watcher() i=7 2024-05-01T13:04:12.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.776+0000 7f10df3a2800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:12.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.778+0000 7f0f92fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:12.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:12.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:12.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 5 note: GC not initialized 2024-05-01T13:04:12.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f0f91fe3640 20 reqs_thread_entry: start 2024-05-01T13:04:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 20 Filter name: none 2024-05-01T13:04:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f0f037fe640 20 reqs_thread_entry: start 2024-05-01T13:04:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.779+0000 7f10df3a2800 20 rados->read ofs=0 len=0 2024-05-01T13:04:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.780+0000 7f10df3a2800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.781+0000 7f0f92fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:12.783 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.783+0000 7f10df3a2800 20 remove_watcher() i=0 2024-05-01T13:04:12.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.783+0000 7f10df3a2800 2 removed watcher, disabling cache 2024-05-01T13:04:12.784 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.785+0000 7f10df3a2800 20 remove_watcher() i=1 2024-05-01T13:04:12.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.786+0000 7f10df3a2800 20 remove_watcher() i=2 2024-05-01T13:04:12.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.787+0000 7f10df3a2800 20 remove_watcher() i=3 2024-05-01T13:04:12.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.788+0000 7f10df3a2800 20 remove_watcher() i=4 2024-05-01T13:04:12.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.789+0000 7f10df3a2800 20 remove_watcher() i=5 2024-05-01T13:04:12.789 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.790+0000 7f10df3a2800 20 remove_watcher() i=6 2024-05-01T13:04:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:12.791+0000 7f10df3a2800 20 remove_watcher() i=7 2024-05-01T13:04:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:14.100 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.150 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.150 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.384 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.384 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.384 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.387 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.390 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.390 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.390 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.430 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.432 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.432 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.432 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.453 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:16.453 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.452 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.453 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.659 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.890+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.892+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.892+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.893+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.893+0000 7f3c28611800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.893+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.893+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.893+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.894+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.894+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.895+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.896+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.896+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 zone MyZone found 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.897+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.898+0000 7f3c28611800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.898+0000 7f3c28611800 20 started sync module instance, tier type = 2024-05-01T13:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.898+0000 7f3c28611800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=0 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=1 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=2 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=3 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=4 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=5 2024-05-01T13:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.908+0000 7f3c28611800 20 add_watcher() i=6 2024-05-01T13:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.910+0000 7f3c28611800 20 add_watcher() i=7 2024-05-01T13:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.910+0000 7f3c28611800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.911+0000 7f3ad9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3c28611800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3c28611800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3c28611800 5 note: GC not initialized 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3ad8fe1640 20 reqs_thread_entry: start 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3c28611800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3c28611800 20 Filter name: none 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.912+0000 7f3a527fc640 20 reqs_thread_entry: start 2024-05-01T13:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.913+0000 7f3c28611800 20 rados->read ofs=0 len=0 2024-05-01T13:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.913+0000 7f3c28611800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.915+0000 7f3ad9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.917+0000 7f3c28611800 20 remove_watcher() i=0 2024-05-01T13:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.917+0000 7f3c28611800 2 removed watcher, disabling cache 2024-05-01T13:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.918+0000 7f3c28611800 20 remove_watcher() i=1 2024-05-01T13:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.919+0000 7f3c28611800 20 remove_watcher() i=2 2024-05-01T13:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.920+0000 7f3c28611800 20 remove_watcher() i=3 2024-05-01T13:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.921+0000 7f3c28611800 20 remove_watcher() i=4 2024-05-01T13:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.922+0000 7f3c28611800 20 remove_watcher() i=5 2024-05-01T13:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.923+0000 7f3c28611800 20 remove_watcher() i=6 2024-05-01T13:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:17.924+0000 7f3c28611800 20 remove_watcher() i=7 2024-05-01T13:04:18.734 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.736 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.737 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.738 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.739 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:18.947 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:23.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.023+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.025+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:23.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.025+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.025+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:23.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.025+0000 7fa95d733800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:23.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.025+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.026+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:23.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.026+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.026+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:23.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.026+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.027+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.028+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:23.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.028+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 zone MyZone found 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:23.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:23.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 started sync module instance, tier type = 2024-05-01T13:04:23.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.029+0000 7fa95d733800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=0 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=1 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=2 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=3 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=4 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=5 2024-05-01T13:04:23.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.041+0000 7fa95d733800 20 add_watcher() i=6 2024-05-01T13:04:23.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.042+0000 7fa95d733800 20 add_watcher() i=7 2024-05-01T13:04:23.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.042+0000 7fa95d733800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.044+0000 7fa80f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 5 note: GC not initialized 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa80e7e4640 20 reqs_thread_entry: start 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 20 Filter name: none 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa77ffff640 20 reqs_thread_entry: start 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.045+0000 7fa95d733800 20 rados->read ofs=0 len=0 2024-05-01T13:04:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.046+0000 7fa95d733800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:23.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.047+0000 7fa80f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:23.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.049+0000 7fa95d733800 20 remove_watcher() i=0 2024-05-01T13:04:23.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.049+0000 7fa95d733800 2 removed watcher, disabling cache 2024-05-01T13:04:23.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.051+0000 7fa95d733800 20 remove_watcher() i=1 2024-05-01T13:04:23.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.052+0000 7fa95d733800 20 remove_watcher() i=2 2024-05-01T13:04:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.053+0000 7fa95d733800 20 remove_watcher() i=3 2024-05-01T13:04:23.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.054+0000 7fa95d733800 20 remove_watcher() i=4 2024-05-01T13:04:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.055+0000 7fa95d733800 20 remove_watcher() i=5 2024-05-01T13:04:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.056+0000 7fa95d733800 20 remove_watcher() i=6 2024-05-01T13:04:23.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:23.057+0000 7fa95d733800 20 remove_watcher() i=7 2024-05-01T13:04:28.153 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.154 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.155 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.156 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.164 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.165 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.166 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.168 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:28.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.370+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.371+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.371+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.372+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.372+0000 7fc036d33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.372+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.372+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.372+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.373+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:28.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.373+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:28.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:28.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:28.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.374+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.375+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:28.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.375+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 zone MyZone found 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:28.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.376+0000 7fc036d33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:28.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.377+0000 7fc036d33800 20 started sync module instance, tier type = 2024-05-01T13:04:28.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.377+0000 7fc036d33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=0 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=1 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=2 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=3 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=4 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=5 2024-05-01T13:04:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.398+0000 7fc036d33800 20 add_watcher() i=6 2024-05-01T13:04:28.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.399+0000 7fc036d33800 20 add_watcher() i=7 2024-05-01T13:04:28.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.399+0000 7fc036d33800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.401+0000 7fbee97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.401+0000 7fc036d33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.401+0000 7fc036d33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.401+0000 7fc036d33800 5 note: GC not initialized 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.402+0000 7fbe63fff640 20 reqs_thread_entry: start 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.402+0000 7fc036d33800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.402+0000 7fc036d33800 20 Filter name: none 2024-05-01T13:04:28.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.402+0000 7fbe61ffb640 20 reqs_thread_entry: start 2024-05-01T13:04:28.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.402+0000 7fc036d33800 20 rados->read ofs=0 len=0 2024-05-01T13:04:28.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.403+0000 7fc036d33800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:28.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.404+0000 7fbee97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:28.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.406+0000 7fc036d33800 20 remove_watcher() i=0 2024-05-01T13:04:28.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.406+0000 7fc036d33800 2 removed watcher, disabling cache 2024-05-01T13:04:28.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.407+0000 7fc036d33800 20 remove_watcher() i=1 2024-05-01T13:04:28.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.408+0000 7fc036d33800 20 remove_watcher() i=2 2024-05-01T13:04:28.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.409+0000 7fc036d33800 20 remove_watcher() i=3 2024-05-01T13:04:28.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.410+0000 7fc036d33800 20 remove_watcher() i=4 2024-05-01T13:04:28.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.411+0000 7fc036d33800 20 remove_watcher() i=5 2024-05-01T13:04:28.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.413+0000 7fc036d33800 20 remove_watcher() i=6 2024-05-01T13:04:28.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:28.414+0000 7fc036d33800 20 remove_watcher() i=7 2024-05-01T13:04:29.165 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.165 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.166 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.166 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.167 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.167 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.167 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.167 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.167 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.168 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.168 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.256 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.256 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.256 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.256 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.459 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.460 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.460 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.460 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.460 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:30.467 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:30] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.300 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.300 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.301 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.312 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.312 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.313 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.313 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.314 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.314 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.315 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.315 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.315 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.316 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.316 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:32.316 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.512+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.514+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.514+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.515+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.515+0000 7f8dff527800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.515+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.515+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.515+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.516+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.516+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.516+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.516+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.517+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.517+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.517+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.517+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 zone MyZone found 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.518+0000 7f8dff527800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 started sync module instance, tier type = 2024-05-01T13:04:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.519+0000 7f8dff527800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=0 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=1 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=2 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=3 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=4 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=5 2024-05-01T13:04:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.530+0000 7f8dff527800 20 add_watcher() i=6 2024-05-01T13:04:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.531+0000 7f8dff527800 20 add_watcher() i=7 2024-05-01T13:04:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.531+0000 7f8dff527800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8cb1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8dff527800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8dff527800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8dff527800 5 note: GC not initialized 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8cb0fe1640 20 reqs_thread_entry: start 2024-05-01T13:04:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8dff527800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8dff527800 20 Filter name: none 2024-05-01T13:04:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.533+0000 7f8c2a7fc640 20 reqs_thread_entry: start 2024-05-01T13:04:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.534+0000 7f8dff527800 20 rados->read ofs=0 len=0 2024-05-01T13:04:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.534+0000 7f8dff527800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.536+0000 7f8cb1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.538+0000 7f8dff527800 20 remove_watcher() i=0 2024-05-01T13:04:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.538+0000 7f8dff527800 2 removed watcher, disabling cache 2024-05-01T13:04:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.539+0000 7f8dff527800 20 remove_watcher() i=1 2024-05-01T13:04:33.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.540+0000 7f8dff527800 20 remove_watcher() i=2 2024-05-01T13:04:33.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.541+0000 7f8dff527800 20 remove_watcher() i=3 2024-05-01T13:04:33.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.542+0000 7f8dff527800 20 remove_watcher() i=4 2024-05-01T13:04:33.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.543+0000 7f8dff527800 20 remove_watcher() i=5 2024-05-01T13:04:33.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.545+0000 7f8dff527800 20 remove_watcher() i=6 2024-05-01T13:04:33.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:33.546+0000 7f8dff527800 20 remove_watcher() i=7 2024-05-01T13:04:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.551 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.604 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.605 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.606 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.812 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:34.813 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:37.761 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:37] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.609 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:38.609 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:38.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.646+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.648+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:38.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.648+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.649+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.649+0000 7f9e37333800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.649+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.649+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.649+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.650+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.650+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.650+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.651+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.651+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.651+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.651+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.651+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.652+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.652+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 zone MyZone found 2024-05-01T13:04:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 started sync module instance, tier type = 2024-05-01T13:04:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.653+0000 7f9e37333800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=0 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=1 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=2 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=3 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=4 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=5 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.664+0000 7f9e37333800 20 add_watcher() i=6 2024-05-01T13:04:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.665+0000 7f9e37333800 20 add_watcher() i=7 2024-05-01T13:04:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.665+0000 7f9e37333800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.667+0000 7f9ce97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9e37333800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9e37333800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9e37333800 5 note: GC not initialized 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9c63fff640 20 reqs_thread_entry: start 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9e37333800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9e37333800 20 Filter name: none 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.668+0000 7f9c61ffb640 20 reqs_thread_entry: start 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.669+0000 7f9e37333800 20 rados->read ofs=0 len=0 2024-05-01T13:04:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.669+0000 7f9e37333800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.670+0000 7f9ce97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:38.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.673+0000 7f9e37333800 20 remove_watcher() i=0 2024-05-01T13:04:38.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.673+0000 7f9e37333800 2 removed watcher, disabling cache 2024-05-01T13:04:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.674+0000 7f9e37333800 20 remove_watcher() i=1 2024-05-01T13:04:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.675+0000 7f9e37333800 20 remove_watcher() i=2 2024-05-01T13:04:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.676+0000 7f9e37333800 20 remove_watcher() i=3 2024-05-01T13:04:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.677+0000 7f9e37333800 20 remove_watcher() i=4 2024-05-01T13:04:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.678+0000 7f9e37333800 20 remove_watcher() i=5 2024-05-01T13:04:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.679+0000 7f9e37333800 20 remove_watcher() i=6 2024-05-01T13:04:38.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:38.680+0000 7f9e37333800 20 remove_watcher() i=7 2024-05-01T13:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:38] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:04:39] "POST / HTTP/1.1" 200 - 2024-05-01T13:04:43.742 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:43.742 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:43.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.780+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.781+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:43.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.781+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.782+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.782+0000 7fa96df2e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.782+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.782+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.782+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.783+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.783+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:43.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.784+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.785+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:43.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.785+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.785+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 zone MyZone found 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 started sync module instance, tier type = 2024-05-01T13:04:43.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.786+0000 7fa96df2e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=0 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=1 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=2 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=3 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=4 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=5 2024-05-01T13:04:43.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.796+0000 7fa96df2e800 20 add_watcher() i=6 2024-05-01T13:04:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.798+0000 7fa96df2e800 20 add_watcher() i=7 2024-05-01T13:04:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.798+0000 7fa96df2e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.799+0000 7fa81b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 5 note: GC not initialized 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa81a7fc640 20 reqs_thread_entry: start 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 20 Filter name: none 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa78bfff640 20 reqs_thread_entry: start 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.800+0000 7fa96df2e800 20 rados->read ofs=0 len=0 2024-05-01T13:04:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.801+0000 7fa96df2e800 20 rados_obj.operate() r=0 bl.length=225 2024-05-01T13:04:43.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.802+0000 7fa81b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:43.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.804+0000 7fa96df2e800 20 remove_watcher() i=0 2024-05-01T13:04:43.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.804+0000 7fa96df2e800 2 removed watcher, disabling cache 2024-05-01T13:04:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.805+0000 7fa96df2e800 20 remove_watcher() i=1 2024-05-01T13:04:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.806+0000 7fa96df2e800 20 remove_watcher() i=2 2024-05-01T13:04:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.807+0000 7fa96df2e800 20 remove_watcher() i=3 2024-05-01T13:04:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.808+0000 7fa96df2e800 20 remove_watcher() i=4 2024-05-01T13:04:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.810+0000 7fa96df2e800 20 remove_watcher() i=5 2024-05-01T13:04:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.811+0000 7fa96df2e800 20 remove_watcher() i=6 2024-05-01T13:04:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:43.812+0000 7fa96df2e800 20 remove_watcher() i=7 2024-05-01T13:04:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:04:48.943 INFO:teuthology.orchestra.run.smithi050.stderr:test pushing persistent notification via http for account user ... ignoring --setuser ceph since I am not root 2024-05-01T13:04:48.943 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:48.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.981+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.982+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:48.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.982+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.983+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:48.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.983+0000 7f61c5d2b800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:48.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.983+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.983+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:48.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.983+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.984+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:48.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.984+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.985+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:48.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.985+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.985+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:48.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.985+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.986+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:48.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.986+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.986+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:48.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.986+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 zone MyZone found 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:48.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.987+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:48.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.988+0000 7f61c5d2b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:48.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.988+0000 7f61c5d2b800 20 started sync module instance, tier type = 2024-05-01T13:04:48.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:48.988+0000 7f61c5d2b800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=0 2024-05-01T13:04:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=1 2024-05-01T13:04:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=2 2024-05-01T13:04:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=3 2024-05-01T13:04:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=4 2024-05-01T13:04:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=5 2024-05-01T13:04:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.000+0000 7f61c5d2b800 20 add_watcher() i=6 2024-05-01T13:04:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.001+0000 7f61c5d2b800 20 add_watcher() i=7 2024-05-01T13:04:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.001+0000 7f61c5d2b800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.003+0000 7f60737fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.003+0000 7f61c5d2b800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.003+0000 7f61c5d2b800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.003+0000 7f61c5d2b800 5 note: GC not initialized 2024-05-01T13:04:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f60727fc640 20 reqs_thread_entry: start 2024-05-01T13:04:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 20 Filter name: none 2024-05-01T13:04:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f5fe3fff640 20 reqs_thread_entry: start 2024-05-01T13:04:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 10 cache get: name=MyZone.rgw.meta+accounts+name.$testacct : miss 2024-05-01T13:04:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 10 cache put: name=MyZone.rgw.meta+accounts+name.$testacct info.flags=0x0 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 10 adding MyZone.rgw.meta+accounts+name.$testacct to cache LRU end 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.004+0000 7f61c5d2b800 20 failed to read name.$testacct with: (2) No such file or directory 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.005+0000 7f61c5d2b800 10 cache put: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 info.flags=0x17 2024-05-01T13:04:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.005+0000 7f61c5d2b800 10 adding MyZone.rgw.meta+accounts+account.RGW77777777777777777 to cache LRU end 2024-05-01T13:04:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.005+0000 7f61c5d2b800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:accounts:account.RGW77777777777777777, ofs0, ns] 2024-05-01T13:04:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.006+0000 7f60a57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757285 cookie 94092879101600 notifier 6368 bl.length()=301 2024-05-01T13:04:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.006+0000 7f60a57fa640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 info.flags=0x17 2024-05-01T13:04:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.006+0000 7f60a57fa640 10 rgw watcher librados: moving MyZone.rgw.meta+accounts+account.RGW77777777777777777 to cache LRU end 2024-05-01T13:04:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f61c5d2b800 10 cache put: name=MyZone.rgw.meta+accounts+name.$testacct info.flags=0x17 2024-05-01T13:04:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f61c5d2b800 10 moving MyZone.rgw.meta+accounts+name.$testacct to cache LRU end 2024-05-01T13:04:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f61c5d2b800 10 distributing notification oid=MyZone.rgw.control:notify.4 cni=[op: 0, obj: MyZone.rgw.meta:accounts:name.$testacct, ofs0, ns] 2024-05-01T13:04:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f60a4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757286 cookie 94092879123744 notifier 6368 bl.length()=209 2024-05-01T13:04:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f60a4ff9640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+accounts+name.$testacct info.flags=0x17 2024-05-01T13:04:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.008+0000 7f60a4ff9640 10 rgw watcher librados: moving MyZone.rgw.meta+accounts+name.$testacct to cache LRU end 2024-05-01T13:04:49.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.010+0000 7f60737fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.012+0000 7f61c5d2b800 20 remove_watcher() i=0 2024-05-01T13:04:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.012+0000 7f61c5d2b800 2 removed watcher, disabling cache 2024-05-01T13:04:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.013+0000 7f61c5d2b800 20 remove_watcher() i=1 2024-05-01T13:04:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.014+0000 7f61c5d2b800 20 remove_watcher() i=2 2024-05-01T13:04:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.015+0000 7f61c5d2b800 20 remove_watcher() i=3 2024-05-01T13:04:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.016+0000 7f61c5d2b800 20 remove_watcher() i=4 2024-05-01T13:04:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.018+0000 7f61c5d2b800 20 remove_watcher() i=5 2024-05-01T13:04:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.019+0000 7f61c5d2b800 20 remove_watcher() i=6 2024-05-01T13:04:49.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.020+0000 7f61c5d2b800 20 remove_watcher() i=7 2024-05-01T13:04:49.086 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:49.086 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:49.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.124+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.126+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:49.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.126+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.127+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:49.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.127+0000 7efe88662800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:49.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.127+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.127+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:49.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.127+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.128+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:49.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.128+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.128+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.128+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.129+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.129+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.129+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.129+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 zone MyZone found 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.130+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.131+0000 7efe88662800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.131+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.131+0000 7efe88662800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.131+0000 7efe88662800 20 started sync module instance, tier type = 2024-05-01T13:04:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.131+0000 7efe88662800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=0 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=1 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=2 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=3 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=4 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=5 2024-05-01T13:04:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.142+0000 7efe88662800 20 add_watcher() i=6 2024-05-01T13:04:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.143+0000 7efe88662800 20 add_watcher() i=7 2024-05-01T13:04:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.143+0000 7efe88662800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.145+0000 7efd3cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.145+0000 7efe88662800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.145+0000 7efe88662800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.145+0000 7efe88662800 5 note: GC not initialized 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efcb77fe640 20 reqs_thread_entry: start 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 Filter name: none 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efcb57fa640 20 reqs_thread_entry: start 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 cache get: name=MyZone.rgw.meta+users.uid+supermantest : miss 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 cache put: name=MyZone.rgw.meta+users.uid+supermantest info.flags=0x0 2024-05-01T13:04:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 adding MyZone.rgw.meta+users.uid+supermantest to cache LRU end 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568689.032559 : miss 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568689.032559 info.flags=0x0 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 adding MyZone.rgw.meta+users.keys+1714568689.032559 to cache LRU end 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 10 cache get: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 : miss 2024-05-01T13:04:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.146+0000 7efe88662800 20 rados->read ofs=0 len=0 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 20 rados_obj.operate() r=0 bl.length=102 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 10 cache put: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 info.flags=0x17 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 10 adding MyZone.rgw.meta+accounts+account.RGW77777777777777777 to cache LRU end 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568689.032559 : hit (negative entry) 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.147+0000 7efe88662800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568689.032559 : hit (negative entry) 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.148+0000 7efe88662800 10 cache put: name=MyZone.rgw.meta+users.uid+supermantest info.flags=0x17 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.148+0000 7efe88662800 10 moving MyZone.rgw.meta+users.uid+supermantest to cache LRU end 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.148+0000 7efe88662800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:supermantest, ofs0, ns] 2024-05-01T13:04:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.149+0000 7efd69ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757287 cookie 94096641610880 notifier 6759 bl.length()=552 2024-05-01T13:04:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.149+0000 7efd69ffb640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+supermantest info.flags=0x17 2024-05-01T13:04:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.149+0000 7efd69ffb640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+supermantest to cache LRU end 2024-05-01T13:04:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efe88662800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568689.032559 info.flags=0x7 2024-05-01T13:04:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efe88662800 10 moving MyZone.rgw.meta+users.keys+1714568689.032559 to cache LRU end 2024-05-01T13:04:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efe88662800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568689.032559, ofs0, ns] 2024-05-01T13:04:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efd697fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757306 cookie 94096641637088 notifier 6759 bl.length()=182 2024-05-01T13:04:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efd697fa640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568689.032559 info.flags=0x7 2024-05-01T13:04:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.150+0000 7efd697fa640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568689.032559 to cache LRU end 2024-05-01T13:04:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.152+0000 7efe88662800 20 linked user supermantest to account RGW77777777777777777 2024-05-01T13:04:49.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.153+0000 7efd3cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:49.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.155+0000 7efe88662800 20 remove_watcher() i=0 2024-05-01T13:04:49.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.155+0000 7efe88662800 2 removed watcher, disabling cache 2024-05-01T13:04:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.156+0000 7efe88662800 20 remove_watcher() i=1 2024-05-01T13:04:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.157+0000 7efe88662800 20 remove_watcher() i=2 2024-05-01T13:04:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.159+0000 7efe88662800 20 remove_watcher() i=3 2024-05-01T13:04:49.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.160+0000 7efe88662800 20 remove_watcher() i=4 2024-05-01T13:04:49.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.161+0000 7efe88662800 20 remove_watcher() i=5 2024-05-01T13:04:49.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.162+0000 7efe88662800 20 remove_watcher() i=6 2024-05-01T13:04:49.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:49.163+0000 7efe88662800 20 remove_watcher() i=7 2024-05-01T13:04:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:51.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.701+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.703+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:51.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.703+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.704+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:51.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.704+0000 7fa41ca13800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:51.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.704+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.704+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:51.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.704+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.705+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:51.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.705+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.705+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:51.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.705+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.706+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:51.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.706+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.706+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.706+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 zone MyZone found 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.707+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.708+0000 7fa41ca13800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:51.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.708+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.708+0000 7fa41ca13800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:51.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.708+0000 7fa41ca13800 20 started sync module instance, tier type = 2024-05-01T13:04:51.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.708+0000 7fa41ca13800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=0 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=1 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=2 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=3 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=4 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=5 2024-05-01T13:04:51.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.721+0000 7fa41ca13800 20 add_watcher() i=6 2024-05-01T13:04:51.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.722+0000 7fa41ca13800 20 add_watcher() i=7 2024-05-01T13:04:51.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.722+0000 7fa41ca13800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:51.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.724+0000 7fa2cefdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.724+0000 7fa41ca13800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.724+0000 7fa41ca13800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.724+0000 7fa41ca13800 5 note: GC not initialized 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.724+0000 7fa2cdfdb640 20 reqs_thread_entry: start 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.725+0000 7fa41ca13800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.725+0000 7fa41ca13800 20 Filter name: none 2024-05-01T13:04:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.725+0000 7fa23f7fe640 20 reqs_thread_entry: start 2024-05-01T13:04:51.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.725+0000 7fa41ca13800 20 rados->read ofs=0 len=0 2024-05-01T13:04:51.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.726+0000 7fa41ca13800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:04:51.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.727+0000 7fa2cefdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.730+0000 7fa41ca13800 20 remove_watcher() i=0 2024-05-01T13:04:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.730+0000 7fa41ca13800 2 removed watcher, disabling cache 2024-05-01T13:04:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.731+0000 7fa41ca13800 20 remove_watcher() i=1 2024-05-01T13:04:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.732+0000 7fa41ca13800 20 remove_watcher() i=2 2024-05-01T13:04:51.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.733+0000 7fa41ca13800 20 remove_watcher() i=3 2024-05-01T13:04:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.734+0000 7fa41ca13800 20 remove_watcher() i=4 2024-05-01T13:04:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.735+0000 7fa41ca13800 20 remove_watcher() i=5 2024-05-01T13:04:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.736+0000 7fa41ca13800 20 remove_watcher() i=6 2024-05-01T13:04:51.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:51.737+0000 7fa41ca13800 20 remove_watcher() i=7 2024-05-01T13:04:56.804 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:04:56.804 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:04:56.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.841+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.843+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.843+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.844+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.844+0000 7f232eb23800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:04:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.844+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.844+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.844+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:04:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:04:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.845+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.846+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:04:56.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.846+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.846+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:04:56.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.846+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:04:56.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 20 zone MyZone found 2024-05-01T13:04:56.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 20 period zonegroup name MyZoneGroup 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.847+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.848+0000 7f232eb23800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.848+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.848+0000 7f232eb23800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.848+0000 7f232eb23800 20 started sync module instance, tier type = 2024-05-01T13:04:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.848+0000 7f232eb23800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:04:56.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=0 2024-05-01T13:04:56.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=1 2024-05-01T13:04:56.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=2 2024-05-01T13:04:56.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=3 2024-05-01T13:04:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=4 2024-05-01T13:04:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=5 2024-05-01T13:04:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.859+0000 7f232eb23800 20 add_watcher() i=6 2024-05-01T13:04:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.860+0000 7f232eb23800 20 add_watcher() i=7 2024-05-01T13:04:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.860+0000 7f232eb23800 2 all 8 watchers are set, enabling cache 2024-05-01T13:04:56.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.862+0000 7f21e17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:56.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.862+0000 7f232eb23800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:04:56.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.862+0000 7f232eb23800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:04:56.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.862+0000 7f232eb23800 5 note: GC not initialized 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.862+0000 7f215bfff640 20 reqs_thread_entry: start 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.863+0000 7f232eb23800 20 init_complete bucket index max shards: 11 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.863+0000 7f232eb23800 20 Filter name: none 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.863+0000 7f2159ffb640 20 reqs_thread_entry: start 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.863+0000 7f232eb23800 20 rados->read ofs=0 len=0 2024-05-01T13:04:56.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.864+0000 7f232eb23800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:04:56.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.865+0000 7f21e17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:04:56.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.868+0000 7f232eb23800 20 remove_watcher() i=0 2024-05-01T13:04:56.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.868+0000 7f232eb23800 2 removed watcher, disabling cache 2024-05-01T13:04:56.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.869+0000 7f232eb23800 20 remove_watcher() i=1 2024-05-01T13:04:56.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.870+0000 7f232eb23800 20 remove_watcher() i=2 2024-05-01T13:04:56.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.871+0000 7f232eb23800 20 remove_watcher() i=3 2024-05-01T13:04:56.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.872+0000 7f232eb23800 20 remove_watcher() i=4 2024-05-01T13:04:56.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.873+0000 7f232eb23800 20 remove_watcher() i=5 2024-05-01T13:04:56.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.874+0000 7f232eb23800 20 remove_watcher() i=6 2024-05-01T13:04:56.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:04:56.875+0000 7f232eb23800 20 remove_watcher() i=7 2024-05-01T13:05:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:01.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.974+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.974+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:01.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.975+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.975+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.975+0000 7f805df33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.975+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.976+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.976+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.976+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.976+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.977+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.977+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.977+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.977+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.978+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.978+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 zone MyZone found 2024-05-01T13:05:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.979+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.980+0000 7f805df33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.980+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.980+0000 7f805df33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.980+0000 7f805df33800 20 started sync module instance, tier type = 2024-05-01T13:05:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.980+0000 7f805df33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=0 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=1 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=2 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=3 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=4 2024-05-01T13:05:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=5 2024-05-01T13:05:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=6 2024-05-01T13:05:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 20 add_watcher() i=7 2024-05-01T13:05:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.992+0000 7f805df33800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:01.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f7ef37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f805df33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f805df33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f805df33800 5 note: GC not initialized 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f7ef27fc640 20 reqs_thread_entry: start 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f805df33800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f805df33800 20 Filter name: none 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.994+0000 7f7e7ffff640 20 reqs_thread_entry: start 2024-05-01T13:05:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.995+0000 7f805df33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.995+0000 7f805df33800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:01.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.997+0000 7f7ef37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.999+0000 7f805df33800 20 remove_watcher() i=0 2024-05-01T13:05:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:01.999+0000 7f805df33800 2 removed watcher, disabling cache 2024-05-01T13:05:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.000+0000 7f805df33800 20 remove_watcher() i=1 2024-05-01T13:05:02.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.001+0000 7f805df33800 20 remove_watcher() i=2 2024-05-01T13:05:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.002+0000 7f805df33800 20 remove_watcher() i=3 2024-05-01T13:05:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.003+0000 7f805df33800 20 remove_watcher() i=4 2024-05-01T13:05:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.004+0000 7f805df33800 20 remove_watcher() i=5 2024-05-01T13:05:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.006+0000 7f805df33800 20 remove_watcher() i=6 2024-05-01T13:05:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:02.007+0000 7f805df33800 20 remove_watcher() i=7 2024-05-01T13:05:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.108+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.110+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.110+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.111+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.111+0000 7f7e27462800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.111+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.111+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.111+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.112+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.112+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.112+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.112+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.113+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.113+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.113+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.113+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.114+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.114+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.114+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.114+0000 7f7e27462800 20 zone MyZone found 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 started sync module instance, tier type = 2024-05-01T13:05:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.115+0000 7f7e27462800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:07.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=0 2024-05-01T13:05:07.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=1 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=2 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=3 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=4 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=5 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.127+0000 7f7e27462800 20 add_watcher() i=6 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.128+0000 7f7e27462800 20 add_watcher() i=7 2024-05-01T13:05:07.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.128+0000 7f7e27462800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:07.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.130+0000 7f7cdafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.130+0000 7f7e27462800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.130+0000 7f7e27462800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.130+0000 7f7e27462800 5 note: GC not initialized 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.131+0000 7f7cd9fe3640 20 reqs_thread_entry: start 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.131+0000 7f7e27462800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.131+0000 7f7e27462800 20 Filter name: none 2024-05-01T13:05:07.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.131+0000 7f7c4b7fe640 20 reqs_thread_entry: start 2024-05-01T13:05:07.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.131+0000 7f7e27462800 20 rados->read ofs=0 len=0 2024-05-01T13:05:07.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.132+0000 7f7e27462800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:07.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.133+0000 7f7cdafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:07.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.136+0000 7f7e27462800 20 remove_watcher() i=0 2024-05-01T13:05:07.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.136+0000 7f7e27462800 2 removed watcher, disabling cache 2024-05-01T13:05:07.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.137+0000 7f7e27462800 20 remove_watcher() i=1 2024-05-01T13:05:07.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.138+0000 7f7e27462800 20 remove_watcher() i=2 2024-05-01T13:05:07.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.139+0000 7f7e27462800 20 remove_watcher() i=3 2024-05-01T13:05:07.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.140+0000 7f7e27462800 20 remove_watcher() i=4 2024-05-01T13:05:07.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.141+0000 7f7e27462800 20 remove_watcher() i=5 2024-05-01T13:05:07.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.142+0000 7f7e27462800 20 remove_watcher() i=6 2024-05-01T13:05:07.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:07.144+0000 7f7e27462800 20 remove_watcher() i=7 2024-05-01T13:05:12.211 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:12.211 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.250+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.252+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:12.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.252+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.253+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:12.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.253+0000 7f159f81f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:12.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.253+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.253+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:12.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.253+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.254+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.254+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.255+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.256+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.256+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 zone MyZone found 2024-05-01T13:05:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:12.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 started sync module instance, tier type = 2024-05-01T13:05:12.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.257+0000 7f159f81f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:12.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=0 2024-05-01T13:05:12.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=1 2024-05-01T13:05:12.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=2 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=3 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=4 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=5 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.268+0000 7f159f81f800 20 add_watcher() i=6 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.269+0000 7f159f81f800 20 add_watcher() i=7 2024-05-01T13:05:12.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.269+0000 7f159f81f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:12.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f14537e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f159f81f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f159f81f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f159f81f800 5 note: GC not initialized 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f14527e4640 20 reqs_thread_entry: start 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.271+0000 7f159f81f800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.272+0000 7f159f81f800 20 Filter name: none 2024-05-01T13:05:12.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.272+0000 7f13c3fff640 20 reqs_thread_entry: start 2024-05-01T13:05:12.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.272+0000 7f159f81f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:12.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.272+0000 7f159f81f800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:12.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.274+0000 7f14537e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:12.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.277+0000 7f159f81f800 20 remove_watcher() i=0 2024-05-01T13:05:12.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.277+0000 7f159f81f800 2 removed watcher, disabling cache 2024-05-01T13:05:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.278+0000 7f159f81f800 20 remove_watcher() i=1 2024-05-01T13:05:12.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.278+0000 7f159f81f800 20 remove_watcher() i=2 2024-05-01T13:05:12.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.280+0000 7f159f81f800 20 remove_watcher() i=3 2024-05-01T13:05:12.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.281+0000 7f159f81f800 20 remove_watcher() i=4 2024-05-01T13:05:12.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.282+0000 7f159f81f800 20 remove_watcher() i=5 2024-05-01T13:05:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.283+0000 7f159f81f800 20 remove_watcher() i=6 2024-05-01T13:05:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:12.284+0000 7f159f81f800 20 remove_watcher() i=7 2024-05-01T13:05:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:12] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:13] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:14] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:15] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:16.029 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:16.029 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:17.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.384+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.386+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.386+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.387+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.387+0000 7f052fd33800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.387+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.387+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.387+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.388+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.388+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.388+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.388+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.389+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.389+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.389+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.389+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 zone MyZone found 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.390+0000 7f052fd33800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 started sync module instance, tier type = 2024-05-01T13:05:17.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.391+0000 7f052fd33800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=0 2024-05-01T13:05:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=1 2024-05-01T13:05:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=2 2024-05-01T13:05:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=3 2024-05-01T13:05:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=4 2024-05-01T13:05:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=5 2024-05-01T13:05:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.402+0000 7f052fd33800 20 add_watcher() i=6 2024-05-01T13:05:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.403+0000 7f052fd33800 20 add_watcher() i=7 2024-05-01T13:05:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.403+0000 7f052fd33800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.405+0000 7f03e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 5 note: GC not initialized 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f03e0fe1640 20 reqs_thread_entry: start 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 20 Filter name: none 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f035a7fc640 20 reqs_thread_entry: start 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.406+0000 7f052fd33800 20 rados->read ofs=0 len=0 2024-05-01T13:05:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.407+0000 7f052fd33800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:17.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.409+0000 7f03e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.411+0000 7f052fd33800 20 remove_watcher() i=0 2024-05-01T13:05:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.411+0000 7f052fd33800 2 removed watcher, disabling cache 2024-05-01T13:05:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.412+0000 7f052fd33800 20 remove_watcher() i=1 2024-05-01T13:05:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.413+0000 7f052fd33800 20 remove_watcher() i=2 2024-05-01T13:05:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.414+0000 7f052fd33800 20 remove_watcher() i=3 2024-05-01T13:05:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.415+0000 7f052fd33800 20 remove_watcher() i=4 2024-05-01T13:05:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.416+0000 7f052fd33800 20 remove_watcher() i=5 2024-05-01T13:05:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.417+0000 7f052fd33800 20 remove_watcher() i=6 2024-05-01T13:05:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:17.418+0000 7f052fd33800 20 remove_watcher() i=7 2024-05-01T13:05:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.893 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.895 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.895 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.896 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.896 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.896 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.896 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.897 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.897 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.897 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.897 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.898 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.898 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.898 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.912 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:19] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:22.491 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.521+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.523+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.523+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.524+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.524+0000 7fc0aa840800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.524+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.524+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.524+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.525+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.525+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.526+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.526+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.526+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.526+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.527+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.527+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.527+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.527+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 zone MyZone found 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.528+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.529+0000 7fc0aa840800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.529+0000 7fc0aa840800 20 started sync module instance, tier type = 2024-05-01T13:05:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.529+0000 7fc0aa840800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=0 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=1 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=2 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=3 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=4 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=5 2024-05-01T13:05:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.540+0000 7fc0aa840800 20 add_watcher() i=6 2024-05-01T13:05:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.542+0000 7fc0aa840800 20 add_watcher() i=7 2024-05-01T13:05:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.542+0000 7fc0aa840800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fbf5b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fc0aa840800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fc0aa840800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fc0aa840800 5 note: GC not initialized 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fbf5a7e4640 20 reqs_thread_entry: start 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fc0aa840800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fc0aa840800 20 Filter name: none 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.544+0000 7fbecbfff640 20 reqs_thread_entry: start 2024-05-01T13:05:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.545+0000 7fc0aa840800 20 rados->read ofs=0 len=0 2024-05-01T13:05:22.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.545+0000 7fc0aa840800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.547+0000 7fbf5b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.549+0000 7fc0aa840800 20 remove_watcher() i=0 2024-05-01T13:05:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.549+0000 7fc0aa840800 2 removed watcher, disabling cache 2024-05-01T13:05:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.550+0000 7fc0aa840800 20 remove_watcher() i=1 2024-05-01T13:05:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.551+0000 7fc0aa840800 20 remove_watcher() i=2 2024-05-01T13:05:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.552+0000 7fc0aa840800 20 remove_watcher() i=3 2024-05-01T13:05:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.553+0000 7fc0aa840800 20 remove_watcher() i=4 2024-05-01T13:05:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.555+0000 7fc0aa840800 20 remove_watcher() i=5 2024-05-01T13:05:22.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.556+0000 7fc0aa840800 20 remove_watcher() i=6 2024-05-01T13:05:22.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:22.557+0000 7fc0aa840800 20 remove_watcher() i=7 2024-05-01T13:05:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.642 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.644 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.647 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.805 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:27.806 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.844+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:27] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:27.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.846+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:27.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.846+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.846+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.846+0000 7f6abde30800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.846+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.847+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.847+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.848+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:27.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.848+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.848+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:27.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.849+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.849+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:27.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.849+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.849+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.849+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.850+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.850+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.850+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.850+0000 7f6abde30800 20 zone MyZone found 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.850+0000 7f6abde30800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:27.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:27.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 started sync module instance, tier type = 2024-05-01T13:05:27.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.851+0000 7f6abde30800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=0 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=1 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=2 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=3 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=4 2024-05-01T13:05:27.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.865+0000 7f6abde30800 20 add_watcher() i=5 2024-05-01T13:05:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.867+0000 7f6abde30800 20 add_watcher() i=6 2024-05-01T13:05:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.867+0000 7f6abde30800 20 add_watcher() i=7 2024-05-01T13:05:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.867+0000 7f6abde30800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.868+0000 7f696f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f6abde30800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f6abde30800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f6abde30800 5 note: GC not initialized 2024-05-01T13:05:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f696e7e4640 20 reqs_thread_entry: start 2024-05-01T13:05:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f6abde30800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f6abde30800 20 Filter name: none 2024-05-01T13:05:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.869+0000 7f68dffff640 20 reqs_thread_entry: start 2024-05-01T13:05:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.870+0000 7f6abde30800 20 rados->read ofs=0 len=0 2024-05-01T13:05:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.870+0000 7f6abde30800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.872+0000 7f696f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:27.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.874+0000 7f6abde30800 20 remove_watcher() i=0 2024-05-01T13:05:27.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.874+0000 7f6abde30800 2 removed watcher, disabling cache 2024-05-01T13:05:27.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.875+0000 7f6abde30800 20 remove_watcher() i=1 2024-05-01T13:05:27.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.876+0000 7f6abde30800 20 remove_watcher() i=2 2024-05-01T13:05:27.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.877+0000 7f6abde30800 20 remove_watcher() i=3 2024-05-01T13:05:27.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.878+0000 7f6abde30800 20 remove_watcher() i=4 2024-05-01T13:05:27.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.879+0000 7f6abde30800 20 remove_watcher() i=5 2024-05-01T13:05:27.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.881+0000 7f6abde30800 20 remove_watcher() i=6 2024-05-01T13:05:27.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:27.882+0000 7f6abde30800 20 remove_watcher() i=7 2024-05-01T13:05:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.691 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.691 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.691 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.691 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.692 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.692 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.710 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.710 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.710 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.713 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.714 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.714 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.714 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.714 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.715 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.715 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.715 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.716 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.716 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.716 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.717 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.717 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:28.717 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:28] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.774 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.775 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.776 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.776 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.777 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.777 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.820 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.820 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.820 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.821 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.821 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.821 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.848 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.848 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.848 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.848 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.849 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.849 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.850 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.850 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.851 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:31.852 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.912 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.912 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.912 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.914 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.914 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.915 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.935 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.937 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.937 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.937 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.941 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.942 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.942 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:32] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:32.948 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:32.948 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:32.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.985+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.987+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:32.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.987+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.988+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:32.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.988+0000 7fb57cb1f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:32.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.988+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.988+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:32.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.988+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.989+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:32.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.989+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:32.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:32.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:32.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.990+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.991+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:32.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.991+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 zone MyZone found 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.992+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.993+0000 7fb57cb1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.993+0000 7fb57cb1f800 20 started sync module instance, tier type = 2024-05-01T13:05:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:32.993+0000 7fb57cb1f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=0 2024-05-01T13:05:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=1 2024-05-01T13:05:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=2 2024-05-01T13:05:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=3 2024-05-01T13:05:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=4 2024-05-01T13:05:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=5 2024-05-01T13:05:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.005+0000 7fb57cb1f800 20 add_watcher() i=6 2024-05-01T13:05:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.007+0000 7fb57cb1f800 20 add_watcher() i=7 2024-05-01T13:05:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.007+0000 7fb57cb1f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.008+0000 7fb42e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb57cb1f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb57cb1f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb57cb1f800 5 note: GC not initialized 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb42d7e2640 20 reqs_thread_entry: start 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb57cb1f800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb57cb1f800 20 Filter name: none 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.009+0000 7fb39effd640 20 reqs_thread_entry: start 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.010+0000 7fb57cb1f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.010+0000 7fb57cb1f800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.012+0000 7fb42e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.014+0000 7fb57cb1f800 20 remove_watcher() i=0 2024-05-01T13:05:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.014+0000 7fb57cb1f800 2 removed watcher, disabling cache 2024-05-01T13:05:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.015+0000 7fb57cb1f800 20 remove_watcher() i=1 2024-05-01T13:05:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.016+0000 7fb57cb1f800 20 remove_watcher() i=2 2024-05-01T13:05:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.017+0000 7fb57cb1f800 20 remove_watcher() i=3 2024-05-01T13:05:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.018+0000 7fb57cb1f800 20 remove_watcher() i=4 2024-05-01T13:05:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.019+0000 7fb57cb1f800 20 remove_watcher() i=5 2024-05-01T13:05:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.020+0000 7fb57cb1f800 20 remove_watcher() i=6 2024-05-01T13:05:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:33.021+0000 7fb57cb1f800 20 remove_watcher() i=7 2024-05-01T13:05:33.150 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.997 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.998 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.998 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:33] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.211 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.240 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.241 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.241 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.241 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.242 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.242 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.242 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:34] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.307 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:35.307 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:35] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.364 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.365 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.365 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.366 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.366 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.366 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.366 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.367 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.367 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.367 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.385 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:172.21.15.63 - - [01/May/2024 13:05:36] "POST / HTTP/1.1" 200 - 2024-05-01T13:05:38.086 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:38.086 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.124+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.126+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.126+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.127+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.127+0000 7f7bc9a12800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.127+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.127+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.127+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.128+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.128+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.128+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.129+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.129+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.129+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.129+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.129+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.130+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.130+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 zone MyZone found 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.131+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.132+0000 7f7bc9a12800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.132+0000 7f7bc9a12800 20 started sync module instance, tier type = 2024-05-01T13:05:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.132+0000 7f7bc9a12800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=0 2024-05-01T13:05:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=1 2024-05-01T13:05:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=2 2024-05-01T13:05:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=3 2024-05-01T13:05:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=4 2024-05-01T13:05:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=5 2024-05-01T13:05:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.141+0000 7f7bc9a12800 20 add_watcher() i=6 2024-05-01T13:05:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.142+0000 7f7bc9a12800 20 add_watcher() i=7 2024-05-01T13:05:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.142+0000 7f7bc9a12800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.144+0000 7f7a7afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 5 note: GC not initialized 2024-05-01T13:05:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7a79fe3640 20 reqs_thread_entry: start 2024-05-01T13:05:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 20 Filter name: none 2024-05-01T13:05:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f79eb7fe640 20 reqs_thread_entry: start 2024-05-01T13:05:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.145+0000 7f7bc9a12800 20 rados->read ofs=0 len=0 2024-05-01T13:05:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.146+0000 7f7bc9a12800 20 rados_obj.operate() r=0 bl.length=265 2024-05-01T13:05:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.148+0000 7f7a7afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.150+0000 7f7bc9a12800 20 remove_watcher() i=0 2024-05-01T13:05:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.150+0000 7f7bc9a12800 2 removed watcher, disabling cache 2024-05-01T13:05:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.151+0000 7f7bc9a12800 20 remove_watcher() i=1 2024-05-01T13:05:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.152+0000 7f7bc9a12800 20 remove_watcher() i=2 2024-05-01T13:05:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.153+0000 7f7bc9a12800 20 remove_watcher() i=3 2024-05-01T13:05:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.154+0000 7f7bc9a12800 20 remove_watcher() i=4 2024-05-01T13:05:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.155+0000 7f7bc9a12800 20 remove_watcher() i=5 2024-05-01T13:05:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.156+0000 7f7bc9a12800 20 remove_watcher() i=6 2024-05-01T13:05:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:38.157+0000 7f7bc9a12800 20 remove_watcher() i=7 2024-05-01T13:05:43.443 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:43.443 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:43.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.481+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.482+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.482+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.483+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.483+0000 7f8d5a41f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:43.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.483+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.484+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.484+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.484+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.484+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.485+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.485+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.485+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.485+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.486+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.486+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.486+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.486+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:43.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 zone MyZone found 2024-05-01T13:05:43.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:43.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.487+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.488+0000 7f8d5a41f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.488+0000 7f8d5a41f800 20 started sync module instance, tier type = 2024-05-01T13:05:43.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.488+0000 7f8d5a41f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:43.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=0 2024-05-01T13:05:43.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=1 2024-05-01T13:05:43.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=2 2024-05-01T13:05:43.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=3 2024-05-01T13:05:43.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=4 2024-05-01T13:05:43.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=5 2024-05-01T13:05:43.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.499+0000 7f8d5a41f800 20 add_watcher() i=6 2024-05-01T13:05:43.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.500+0000 7f8d5a41f800 20 add_watcher() i=7 2024-05-01T13:05:43.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.500+0000 7f8d5a41f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:43.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.502+0000 7f8c0dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.502+0000 7f8d5a41f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:43.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.502+0000 7f8d5a41f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8c0cfe1640 20 reqs_thread_entry: start 2024-05-01T13:05:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8d5a41f800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8d5a41f800 20 Filter name: none 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8b867fc640 20 reqs_thread_entry: start 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8d5a41f800 10 cache get: name=MyZone.rgw.meta+users.uid+supermantest : miss 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.538+0000 7f8d5a41f800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.539+0000 7f8d5a41f800 20 rados_obj.operate() r=0 bl.length=368 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.539+0000 7f8d5a41f800 10 cache put: name=MyZone.rgw.meta+users.uid+supermantest info.flags=0x13 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.539+0000 7f8d5a41f800 10 adding MyZone.rgw.meta+users.uid+supermantest to cache LRU end 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.539+0000 7f8d5a41f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T13:05:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.539+0000 7f8d5a41f800 10 removing key index: 1714568689.032559 2024-05-01T13:05:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.540+0000 7f8d5a41f800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 1, obj: MyZone.rgw.meta:users.keys:1714568689.032559, ofs0, ns] 2024-05-01T13:05:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.540+0000 7f8c36ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757309 cookie 94505516506992 notifier 6957 bl.length()=166 2024-05-01T13:05:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.541+0000 7f8d5a41f800 10 removing email index: 2024-05-01T13:05:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.541+0000 7f8d5a41f800 10 removing user index: supermantest 2024-05-01T13:05:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.542+0000 7f8d5a41f800 10 removing MyZone.rgw.meta+users.uid+supermantest from cache 2024-05-01T13:05:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.542+0000 7f8d5a41f800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:users.uid:supermantest, ofs0, ns] 2024-05-01T13:05:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.542+0000 7f8c36ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757288 cookie 94505516481856 notifier 6957 bl.length()=160 2024-05-01T13:05:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.544+0000 7f8c0dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.546+0000 7f8d5a41f800 20 remove_watcher() i=0 2024-05-01T13:05:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.546+0000 7f8d5a41f800 2 removed watcher, disabling cache 2024-05-01T13:05:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.547+0000 7f8d5a41f800 20 remove_watcher() i=1 2024-05-01T13:05:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.548+0000 7f8d5a41f800 20 remove_watcher() i=2 2024-05-01T13:05:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.549+0000 7f8d5a41f800 20 remove_watcher() i=3 2024-05-01T13:05:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.550+0000 7f8d5a41f800 20 remove_watcher() i=4 2024-05-01T13:05:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.551+0000 7f8d5a41f800 20 remove_watcher() i=5 2024-05-01T13:05:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.553+0000 7f8d5a41f800 20 remove_watcher() i=6 2024-05-01T13:05:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.554+0000 7f8d5a41f800 20 remove_watcher() i=7 2024-05-01T13:05:43.611 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:43.611 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:43.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.648+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.649+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.649+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.650+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.650+0000 7fb6c930c800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.650+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.651+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.651+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.651+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.651+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.652+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.652+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.652+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.652+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.653+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.653+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.653+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.653+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:43.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 zone MyZone found 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.654+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.655+0000 7fb6c930c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.655+0000 7fb6c930c800 20 started sync module instance, tier type = 2024-05-01T13:05:43.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.655+0000 7fb6c930c800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=0 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=1 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=2 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=3 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=4 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=5 2024-05-01T13:05:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.665+0000 7fb6c930c800 20 add_watcher() i=6 2024-05-01T13:05:43.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.666+0000 7fb6c930c800 20 add_watcher() i=7 2024-05-01T13:05:43.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.666+0000 7fb6c930c800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:43.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.668+0000 7fb57d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:43.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 5 note: GC not initialized 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb4f7fff640 20 reqs_thread_entry: start 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 20 Filter name: none 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb4f5ffb640 20 reqs_thread_entry: start 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 10 cache get: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 : miss 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.669+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 20 rados_obj.operate() r=0 bl.length=102 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 cache put: name=MyZone.rgw.meta+accounts+account.RGW77777777777777777 info.flags=0x17 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 adding MyZone.rgw.meta+accounts+account.RGW77777777777777777 to cache LRU end 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 cache get: name=MyZone.rgw.meta+users.uid+supermantest : miss 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 cache put: name=MyZone.rgw.meta+users.uid+supermantest info.flags=0x0 2024-05-01T13:05:43.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.670+0000 7fb6c930c800 10 adding MyZone.rgw.meta+users.uid+supermantest to cache LRU end 2024-05-01T13:05:43.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.682+0000 7fb6c930c800 10 removing MyZone.rgw.meta+accounts+account.RGW77777777777777777 from cache 2024-05-01T13:05:43.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.682+0000 7fb6c930c800 10 distributing notification oid=MyZone.rgw.control:notify.1 cni=[op: 1, obj: MyZone.rgw.meta:accounts:account.RGW77777777777777777, ofs0, ns] 2024-05-01T13:05:43.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.682+0000 7fb5aa7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757289 cookie 93912292060240 notifier 6587 bl.length()=175 2024-05-01T13:05:43.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.684+0000 7fb6c930c800 10 distributing notification oid=MyZone.rgw.control:notify.4 cni=[op: 1, obj: MyZone.rgw.meta:accounts:name.$testacct, ofs0, ns] 2024-05-01T13:05:43.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.684+0000 7fb5a9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757290 cookie 93912292082336 notifier 6587 bl.length()=161 2024-05-01T13:05:43.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.685+0000 7fb6c930c800 10 distributing notification oid=MyZone.rgw.control:notify.7 cni=[op: 1, obj: MyZone.rgw.meta:accounts:users.RGW77777777777777777, ofs0, ns] 2024-05-01T13:05:43.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.686+0000 7fb5aa7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757310 cookie 93912292103360 notifier 6587 bl.length()=173 2024-05-01T13:05:43.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.687+0000 7fb57d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.689+0000 7fb6c930c800 20 remove_watcher() i=0 2024-05-01T13:05:43.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.689+0000 7fb6c930c800 2 removed watcher, disabling cache 2024-05-01T13:05:43.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.690+0000 7fb6c930c800 20 remove_watcher() i=1 2024-05-01T13:05:43.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.691+0000 7fb6c930c800 20 remove_watcher() i=2 2024-05-01T13:05:43.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.692+0000 7fb6c930c800 20 remove_watcher() i=3 2024-05-01T13:05:43.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.693+0000 7fb6c930c800 20 remove_watcher() i=4 2024-05-01T13:05:43.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.694+0000 7fb6c930c800 20 remove_watcher() i=5 2024-05-01T13:05:43.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.696+0000 7fb6c930c800 20 remove_watcher() i=6 2024-05-01T13:05:43.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.697+0000 7fb6c930c800 20 remove_watcher() i=7 2024-05-01T13:05:43.711 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:05:43.763 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 topic set/get/delete permissions ... ignoring --setuser ceph since I am not root 2024-05-01T13:05:43.763 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.801+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.802+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.802+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.803+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.803+0000 7fc62f533800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.803+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.803+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.804+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.804+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.804+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.805+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.806+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.806+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 zone MyZone found 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.807+0000 7fc62f533800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.808+0000 7fc62f533800 20 started sync module instance, tier type = 2024-05-01T13:05:43.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.808+0000 7fc62f533800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:43.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=0 2024-05-01T13:05:43.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=1 2024-05-01T13:05:43.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=2 2024-05-01T13:05:43.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=3 2024-05-01T13:05:43.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=4 2024-05-01T13:05:43.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=5 2024-05-01T13:05:43.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.819+0000 7fc62f533800 20 add_watcher() i=6 2024-05-01T13:05:43.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.820+0000 7fc62f533800 20 add_watcher() i=7 2024-05-01T13:05:43.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.820+0000 7fc62f533800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc4e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 5 note: GC not initialized 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc4e0fe1640 20 reqs_thread_entry: start 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 20 Filter name: none 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc45a7fc640 20 reqs_thread_entry: start 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 10 cache get: name=MyZone.rgw.meta+users.uid+superman1714568743.7136 : miss 2024-05-01T13:05:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.822+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 cache put: name=MyZone.rgw.meta+users.uid+superman1714568743.7136 info.flags=0x0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 adding MyZone.rgw.meta+users.uid+superman1714568743.7136 to cache LRU end 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568743.7135923 : miss 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 20 rados->read ofs=0 len=0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568743.7135923 info.flags=0x0 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 adding MyZone.rgw.meta+users.keys+1714568743.7135923 to cache LRU end 2024-05-01T13:05:43.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568743.7135923 : hit (negative entry) 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.823+0000 7fc62f533800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568743.7135923 : hit (negative entry) 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.824+0000 7fc62f533800 10 cache put: name=MyZone.rgw.meta+users.uid+superman1714568743.7136 info.flags=0x17 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.824+0000 7fc62f533800 10 moving MyZone.rgw.meta+users.uid+superman1714568743.7136 to cache LRU end 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.824+0000 7fc62f533800 10 distributing notification oid=MyZone.rgw.control:notify.6 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:superman1714568743.7136, ofs0, ns] 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.825+0000 7fc50effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757311 cookie 94894442478160 notifier 6996 bl.length()=570 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.825+0000 7fc50effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+superman1714568743.7136 info.flags=0x17 2024-05-01T13:05:43.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.825+0000 7fc50effd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+superman1714568743.7136 to cache LRU end 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc62f533800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568743.7135923 info.flags=0x7 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc62f533800 10 moving MyZone.rgw.meta+users.keys+1714568743.7135923 to cache LRU end 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc62f533800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568743.7135923, ofs0, ns] 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc50e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757280 cookie 94894442427472 notifier 6996 bl.length()=194 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc50e7fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568743.7135923 info.flags=0x7 2024-05-01T13:05:43.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.827+0000 7fc50e7fc640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568743.7135923 to cache LRU end 2024-05-01T13:05:43.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.828+0000 7fc4e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:43.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.830+0000 7fc62f533800 20 remove_watcher() i=0 2024-05-01T13:05:43.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.830+0000 7fc62f533800 2 removed watcher, disabling cache 2024-05-01T13:05:43.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.832+0000 7fc62f533800 20 remove_watcher() i=1 2024-05-01T13:05:43.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.833+0000 7fc62f533800 20 remove_watcher() i=2 2024-05-01T13:05:43.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.834+0000 7fc62f533800 20 remove_watcher() i=3 2024-05-01T13:05:43.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.835+0000 7fc62f533800 20 remove_watcher() i=4 2024-05-01T13:05:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.836+0000 7fc62f533800 20 remove_watcher() i=5 2024-05-01T13:05:43.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.837+0000 7fc62f533800 20 remove_watcher() i=6 2024-05-01T13:05:43.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:43.839+0000 7fc62f533800 20 remove_watcher() i=7 2024-05-01T13:05:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:05:44.178 INFO:teuthology.orchestra.run.smithi050.stderr:test s3 topic set/get/delete permissions ... ignoring --setuser ceph since I am not root 2024-05-01T13:05:44.178 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.217+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.219+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.219+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.220+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.220+0000 7f5ab0011800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.220+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.220+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.221+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.221+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:44.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.221+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.222+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:44.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.222+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.222+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.222+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.223+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.223+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.224+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.224+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 zone MyZone found 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.225+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.226+0000 7f5ab0011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.226+0000 7f5ab0011800 20 started sync module instance, tier type = 2024-05-01T13:05:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.226+0000 7f5ab0011800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=0 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=1 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=2 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=3 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=4 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=5 2024-05-01T13:05:44.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.238+0000 7f5ab0011800 20 add_watcher() i=6 2024-05-01T13:05:44.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.239+0000 7f5ab0011800 20 add_watcher() i=7 2024-05-01T13:05:44.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.239+0000 7f5ab0011800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:44.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.241+0000 7f5961fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 5 note: GC not initialized 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5960fe1640 20 reqs_thread_entry: start 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 20 Filter name: none 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f58da7fc640 20 reqs_thread_entry: start 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 10 cache get: name=MyZone.rgw.meta+users.uid+superman1714568744.1266687 : miss 2024-05-01T13:05:44.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.242+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 cache put: name=MyZone.rgw.meta+users.uid+superman1714568744.1266687 info.flags=0x0 2024-05-01T13:05:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 adding MyZone.rgw.meta+users.uid+superman1714568744.1266687 to cache LRU end 2024-05-01T13:05:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568744.1266594 : miss 2024-05-01T13:05:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568744.1266594 info.flags=0x0 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 adding MyZone.rgw.meta+users.keys+1714568744.1266594 to cache LRU end 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568744.1266594 : hit (negative entry) 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.243+0000 7f5ab0011800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568744.1266594 : hit (negative entry) 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.244+0000 7f5ab0011800 10 cache put: name=MyZone.rgw.meta+users.uid+superman1714568744.1266687 info.flags=0x17 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.244+0000 7f5ab0011800 10 moving MyZone.rgw.meta+users.uid+superman1714568744.1266687 to cache LRU end 2024-05-01T13:05:44.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.244+0000 7f5ab0011800 10 distributing notification oid=MyZone.rgw.control:notify.7 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:superman1714568744.1266687, ofs0, ns] 2024-05-01T13:05:44.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.245+0000 7f598effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757316 cookie 94568255906304 notifier 6617 bl.length()=579 2024-05-01T13:05:44.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.245+0000 7f598effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+superman1714568744.1266687 info.flags=0x17 2024-05-01T13:05:44.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.245+0000 7f598effd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+superman1714568744.1266687 to cache LRU end 2024-05-01T13:05:44.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f5ab0011800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568744.1266594 info.flags=0x7 2024-05-01T13:05:44.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f5ab0011800 10 moving MyZone.rgw.meta+users.keys+1714568744.1266594 to cache LRU end 2024-05-01T13:05:44.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f5ab0011800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568744.1266594, ofs0, ns] 2024-05-01T13:05:44.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f598e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757283 cookie 94568255855216 notifier 6617 bl.length()=197 2024-05-01T13:05:44.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f598e7fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568744.1266594 info.flags=0x7 2024-05-01T13:05:44.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.247+0000 7f598e7fc640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568744.1266594 to cache LRU end 2024-05-01T13:05:44.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.248+0000 7f5961fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:44.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.251+0000 7f5ab0011800 20 remove_watcher() i=0 2024-05-01T13:05:44.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.251+0000 7f5ab0011800 2 removed watcher, disabling cache 2024-05-01T13:05:44.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.252+0000 7f5ab0011800 20 remove_watcher() i=1 2024-05-01T13:05:44.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.253+0000 7f5ab0011800 20 remove_watcher() i=2 2024-05-01T13:05:44.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.254+0000 7f5ab0011800 20 remove_watcher() i=3 2024-05-01T13:05:44.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.255+0000 7f5ab0011800 20 remove_watcher() i=4 2024-05-01T13:05:44.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.256+0000 7f5ab0011800 20 remove_watcher() i=5 2024-05-01T13:05:44.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.257+0000 7f5ab0011800 20 remove_watcher() i=6 2024-05-01T13:05:44.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.258+0000 7f5ab0011800 20 remove_watcher() i=7 2024-05-01T13:05:44.445 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:05:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:test data path v2 persistent migration ... ignoring --setuser ceph since I am not root 2024-05-01T13:05:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.518+0000 7f752c51f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.518+0000 7f752c51f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.522+0000 7f736dffb640 20 reqs_thread_entry: start 2024-05-01T13:05:44.580 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:44.580 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:44.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.608+0000 7feb2132e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:44.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.608+0000 7feb2132e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:44.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.612+0000 7fe95b7fe640 20 reqs_thread_entry: start 2024-05-01T13:05:44.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.616+0000 7feb2132e800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:44.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.630+0000 7feb2132e800 10 found existing latest_epoch 1 >= given epoch 1, returning r=-17 2024-05-01T13:05:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:44.685 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.724+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.725+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.725+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.725+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.725+0000 7f6ce2c62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.725+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.726+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.726+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.726+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:44.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.726+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.727+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=1104 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.727+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.727+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.727+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.728+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:44.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.728+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.728+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:44.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.728+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 zone MyZone found 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:1 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:44.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.729+0000 7f6ce2c62800 20 rados->read ofs=0 len=0 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.730+0000 7f6ce2c62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.730+0000 7f6ce2c62800 20 started sync module instance, tier type = 2024-05-01T13:05:44.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.730+0000 7f6ce2c62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=0 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=1 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=2 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=3 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=4 2024-05-01T13:05:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.740+0000 7f6ce2c62800 20 add_watcher() i=5 2024-05-01T13:05:44.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.741+0000 7f6ce2c62800 20 add_watcher() i=6 2024-05-01T13:05:44.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.741+0000 7f6ce2c62800 20 add_watcher() i=7 2024-05-01T13:05:44.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.741+0000 7f6ce2c62800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:44.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.743+0000 7f6b96fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6ce2c62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6ce2c62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6ce2c62800 5 note: GC not initialized 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6b95fdb640 20 reqs_thread_entry: start 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6ce2c62800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6ce2c62800 20 Filter name: none 2024-05-01T13:05:44.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.744+0000 7f6b077fe640 20 reqs_thread_entry: start 2024-05-01T13:05:44.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.746+0000 7f6ce2c62800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:44.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.748+0000 7f6ce2c62800 20 updating latest_epoch from 1 -> 2 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:44.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.753+0000 7f6ce2c62800 4 Committed new epoch 2 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:44.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.755+0000 7f6b96fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:44.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.758+0000 7f6ce2c62800 20 remove_watcher() i=0 2024-05-01T13:05:44.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.758+0000 7f6ce2c62800 2 removed watcher, disabling cache 2024-05-01T13:05:44.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.759+0000 7f6ce2c62800 20 remove_watcher() i=1 2024-05-01T13:05:44.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.760+0000 7f6ce2c62800 20 remove_watcher() i=2 2024-05-01T13:05:44.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.761+0000 7f6ce2c62800 20 remove_watcher() i=3 2024-05-01T13:05:44.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.762+0000 7f6ce2c62800 20 remove_watcher() i=4 2024-05-01T13:05:44.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.764+0000 7f6ce2c62800 20 remove_watcher() i=5 2024-05-01T13:05:44.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.765+0000 7f6ce2c62800 20 remove_watcher() i=6 2024-05-01T13:05:44.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:44.766+0000 7f6ce2c62800 20 remove_watcher() i=7 2024-05-01T13:05:44.837 INFO:tasks.rgw.client.1.smithi063.stdout:*** Caught signal (Segmentation fault) ** 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: in thread 7f46c5118640 thread_name:safe_timer 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: ceph version 19.0.0-3122-gabc5b100 (abc5b10088b379036a15386486ef8fee07d8cc3c) squid (dev) 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47f963e6f0] 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: 2: radosgw(+0x8020cc) [0x5610766230cc] 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: 3: radosgw(+0x313050) [0x561076134050] 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: 4: (void boost::context::detail::context_entry, spawn::detail::spawn_helper, rgw::notify::Manager::process_queue(std::__cxx11::basic_string, std::allocator > const&, spawn::basic_yield_context >)::{lambda(spawn::basic_yield_context >)#7}, boost::context::basic_protected_fixedsize_stack >::operator()()::{lambda(boost::context::continuation&&)#1}> >(boost::context::detail::transfer_t)+0x55) [0x561076632945] 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout: 5: make_fcontext() 2024-05-01T13:05:44.838 INFO:tasks.rgw.client.1.smithi063.stdout:2024-05-01T13:05:44.837+0000 7f46c5118640 -1 *** Caught signal (Segmentation fault) ** 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: in thread 7f46c5118640 thread_name:safe_timer 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: ceph version 19.0.0-3122-gabc5b100 (abc5b10088b379036a15386486ef8fee07d8cc3c) squid (dev) 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47f963e6f0] 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 2: radosgw(+0x8020cc) [0x5610766230cc] 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 3: radosgw(+0x313050) [0x561076134050] 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 4: (void boost::context::detail::context_entry, spawn::detail::spawn_helper, rgw::notify::Manager::process_queue(std::__cxx11::basic_string, std::allocator > const&, spawn::basic_yield_context >)::{lambda(spawn::basic_yield_context >)#7}, boost::context::basic_protected_fixedsize_stack >::operator()()::{lambda(boost::context::continuation&&)#1}> >(boost::context::detail::transfer_t)+0x55) [0x561076632945] 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: 5: make_fcontext() 2024-05-01T13:05:44.839 INFO:tasks.rgw.client.1.smithi063.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-01T13:05:44.840 INFO:tasks.rgw.client.1.smithi063.stdout: 2024-05-01T13:05:44.850 INFO:tasks.rgw.client.1.smithi063.stdout: 0> 2024-05-01T13:05:44.837+0000 7f46c5118640 -1 *** Caught signal (Segmentation fault) ** 2024-05-01T13:05:44.850 INFO:tasks.rgw.client.1.smithi063.stdout: in thread 7f46c5118640 thread_name:safe_timer 2024-05-01T13:05:44.850 INFO:tasks.rgw.client.1.smithi063.stdout: 2024-05-01T13:05:44.850 INFO:tasks.rgw.client.1.smithi063.stdout: ceph version 19.0.0-3122-gabc5b100 (abc5b10088b379036a15386486ef8fee07d8cc3c) squid (dev) 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f47f963e6f0] 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 2: radosgw(+0x8020cc) [0x5610766230cc] 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 3: radosgw(+0x313050) [0x561076134050] 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 4: (void boost::context::detail::context_entry, spawn::detail::spawn_helper, rgw::notify::Manager::process_queue(std::__cxx11::basic_string, std::allocator > const&, spawn::basic_yield_context >)::{lambda(spawn::basic_yield_context >)#7}, boost::context::basic_protected_fixedsize_stack >::operator()()::{lambda(boost::context::continuation&&)#1}> >(boost::context::detail::transfer_t)+0x55) [0x561076632945] 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 5: make_fcontext() 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-01T13:05:44.851 INFO:tasks.rgw.client.1.smithi063.stdout: 2024-05-01T13:05:44.964 INFO:tasks.rgw.client.1.smithi063.stderr:daemon-helper: command crashed with signal 11 2024-05-01T13:05:45.011 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:45.010+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:05:45.011 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:45.010+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:05:45.011 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:45.010+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:05:45.011 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:45.010+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:05:45.123 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:45.123 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.167+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.169+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.169+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.170+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.170+0000 7f44bea40800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.170+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.170+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.170+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.171+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.171+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:05:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.172+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.173+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.173+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 zone MyZone found 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:2 2024-05-01T13:05:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.174+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.175+0000 7f44bea40800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.175+0000 7f44bea40800 20 started sync module instance, tier type = 2024-05-01T13:05:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.175+0000 7f44bea40800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=0 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=1 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=2 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=3 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=4 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=5 2024-05-01T13:05:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.185+0000 7f44bea40800 20 add_watcher() i=6 2024-05-01T13:05:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.186+0000 7f44bea40800 20 add_watcher() i=7 2024-05-01T13:05:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.186+0000 7f44bea40800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.188+0000 7f436f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:45.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 5 note: GC not initialized 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f436e7e4640 20 reqs_thread_entry: start 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 20 Filter name: none 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f42dffff640 20 reqs_thread_entry: start 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.189+0000 7f44bea40800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.190+0000 7f44bea40800 20 rados_obj.operate() r=0 bl.length=524 2024-05-01T13:05:45.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.191+0000 7f436f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.193+0000 7f44bea40800 20 remove_watcher() i=0 2024-05-01T13:05:45.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.193+0000 7f44bea40800 2 removed watcher, disabling cache 2024-05-01T13:05:45.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.194+0000 7f44bea40800 20 remove_watcher() i=1 2024-05-01T13:05:45.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.195+0000 7f44bea40800 20 remove_watcher() i=2 2024-05-01T13:05:45.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.196+0000 7f44bea40800 20 remove_watcher() i=3 2024-05-01T13:05:45.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.197+0000 7f44bea40800 20 remove_watcher() i=4 2024-05-01T13:05:45.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.198+0000 7f44bea40800 20 remove_watcher() i=5 2024-05-01T13:05:45.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.199+0000 7f44bea40800 20 remove_watcher() i=6 2024-05-01T13:05:45.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.200+0000 7f44bea40800 20 remove_watcher() i=7 2024-05-01T13:05:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:45.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.539+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.541+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.541+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.542+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.542+0000 7f57d3923800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:45.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.542+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.542+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.542+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.543+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.543+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.543+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.543+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.544+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.544+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.544+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.544+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 zone MyZone found 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:2 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.545+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.546+0000 7f57d3923800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.546+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.546+0000 7f57d3923800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.546+0000 7f57d3923800 20 started sync module instance, tier type = 2024-05-01T13:05:45.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.546+0000 7f57d3923800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=0 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=1 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=2 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=3 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=4 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=5 2024-05-01T13:05:45.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.559+0000 7f57d3923800 20 add_watcher() i=6 2024-05-01T13:05:45.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.560+0000 7f57d3923800 20 add_watcher() i=7 2024-05-01T13:05:45.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.560+0000 7f57d3923800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:45.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.562+0000 7f5685fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:45.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.562+0000 7f57d3923800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:45.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.562+0000 7f57d3923800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:45.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f57d3923800 5 note: GC not initialized 2024-05-01T13:05:45.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f5684fe1640 20 reqs_thread_entry: start 2024-05-01T13:05:45.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f57d3923800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:45.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f57d3923800 20 Filter name: none 2024-05-01T13:05:45.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f55fe7fc640 20 reqs_thread_entry: start 2024-05-01T13:05:45.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f57d3923800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.563+0000 7f57d3923800 20 rados_obj.operate() r=0 bl.length=524 2024-05-01T13:05:45.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.565+0000 7f5685fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:45.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.567+0000 7f57d3923800 20 remove_watcher() i=0 2024-05-01T13:05:45.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.567+0000 7f57d3923800 2 removed watcher, disabling cache 2024-05-01T13:05:45.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.568+0000 7f57d3923800 20 remove_watcher() i=1 2024-05-01T13:05:45.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.569+0000 7f57d3923800 20 remove_watcher() i=2 2024-05-01T13:05:45.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.570+0000 7f57d3923800 20 remove_watcher() i=3 2024-05-01T13:05:45.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.571+0000 7f57d3923800 20 remove_watcher() i=4 2024-05-01T13:05:45.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.572+0000 7f57d3923800 20 remove_watcher() i=5 2024-05-01T13:05:45.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.573+0000 7f57d3923800 20 remove_watcher() i=6 2024-05-01T13:05:45.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.574+0000 7f57d3923800 20 remove_watcher() i=7 2024-05-01T13:05:45.632 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:45.632 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:45.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.659+0000 7f7b2f11f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:45.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.659+0000 7f7b2f11f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:45.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.663+0000 7f79ed7e2640 20 reqs_thread_entry: start 2024-05-01T13:05:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:45.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.748+0000 7fe09f346800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:45.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.748+0000 7fe09f346800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:45.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.752+0000 7fdee2ffd640 20 reqs_thread_entry: start 2024-05-01T13:05:45.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.755+0000 7fe09f346800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:45.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.767+0000 7fe09f346800 20 updating latest_epoch from 1 -> 2 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:05:45.821 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:45.822 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:45.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.858+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.859+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.859+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.860+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.860+0000 7fca55240800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:45.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.860+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.860+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.860+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.861+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:45.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.861+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.861+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:05:45.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.861+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.862+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.862+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.862+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:45.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.862+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.863+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:45.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.863+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 zone MyZone found 2024-05-01T13:05:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:2 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.864+0000 7fca55240800 20 rados->read ofs=0 len=0 2024-05-01T13:05:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.865+0000 7fca55240800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.865+0000 7fca55240800 20 started sync module instance, tier type = 2024-05-01T13:05:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.865+0000 7fca55240800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=0 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=1 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=2 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=3 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=4 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=5 2024-05-01T13:05:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.875+0000 7fca55240800 20 add_watcher() i=6 2024-05-01T13:05:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.876+0000 7fca55240800 20 add_watcher() i=7 2024-05-01T13:05:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.876+0000 7fca55240800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.877+0000 7fc905fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fca55240800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fca55240800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fca55240800 5 note: GC not initialized 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fc904fe1640 20 reqs_thread_entry: start 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fca55240800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fca55240800 20 Filter name: none 2024-05-01T13:05:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.878+0000 7fc87e7fc640 20 reqs_thread_entry: start 2024-05-01T13:05:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.880+0000 7fca55240800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.882+0000 7fca55240800 20 updating latest_epoch from 2 -> 3 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:45.891+0000 7fca55240800 4 Committed new epoch 3 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:05:46.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.053+0000 7fc92effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757290 cookie 93909319119248 notifier 7188 bl.length()=224 2024-05-01T13:05:46.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.053+0000 7fc92effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.log++meta.history info.flags=0x17 2024-05-01T13:05:46.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.053+0000 7fc92effd640 10 rgw watcher librados: adding MyZone.rgw.log++meta.history to cache LRU end 2024-05-01T13:05:46.091 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:46.091+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:05:46.091 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:46.091+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:05:46.091 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:46.091+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:05:46.091 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:05:46.091+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:05:46.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.112+0000 7fc92e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757321 cookie 93909319159088 notifier 7188 bl.length()=1275 2024-05-01T13:05:46.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.112+0000 7fc92e7fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+root+.bucket.meta.ashubx-22:b3b7ffad-8c5b-4f40-9e07-35170fac4823.6677.1 info.flags=0x17 2024-05-01T13:05:46.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.112+0000 7fc92e7fc640 10 rgw watcher librados: adding MyZone.rgw.meta+root+.bucket.meta.ashubx-22:b3b7ffad-8c5b-4f40-9e07-35170fac4823.6677.1 to cache LRU end 2024-05-01T13:05:46.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.112+0000 7fc92e7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2024-05-01T13:05:46.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.112+0000 7fc92e7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.bucket-notification bl.length()=400 2024-05-01T13:05:46.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.114+0000 7fc92effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757322 cookie 93909319143392 notifier 7188 bl.length()=206 2024-05-01T13:05:46.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.117+0000 7fc92e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757323 cookie 93909319123872 notifier 7188 bl.length()=413 2024-05-01T13:05:46.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.117+0000 7fc92e7fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-22_topic info.flags=0x17 2024-05-01T13:05:46.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.117+0000 7fc92e7fc640 10 rgw watcher librados: adding MyZone.rgw.meta+topics+topic.:ashubx-22_topic to cache LRU end 2024-05-01T13:05:46.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:46.119+0000 7fc92effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757289 cookie 93909319110400 notifier 7188 bl.length()=145 2024-05-01T13:05:48.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~0s 2024-05-01T13:05:55.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~7s 2024-05-01T13:05:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.893+0000 7fc905fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.897+0000 7fca55240800 20 remove_watcher() i=0 2024-05-01T13:05:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.897+0000 7fca55240800 2 removed watcher, disabling cache 2024-05-01T13:05:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.898+0000 7fca55240800 20 remove_watcher() i=1 2024-05-01T13:05:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.899+0000 7fca55240800 20 remove_watcher() i=2 2024-05-01T13:05:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.901+0000 7fca55240800 20 remove_watcher() i=3 2024-05-01T13:05:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.902+0000 7fca55240800 20 remove_watcher() i=4 2024-05-01T13:05:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.903+0000 7fca55240800 20 remove_watcher() i=5 2024-05-01T13:05:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.905+0000 7fca55240800 20 remove_watcher() i=6 2024-05-01T13:05:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:55.907+0000 7fca55240800 20 remove_watcher() i=7 2024-05-01T13:05:56.973 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:56.973 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:57.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.012+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.014+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.014+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.015+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.015+0000 7fe237098800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:57.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.015+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.015+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.015+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.016+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:57.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.016+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:05:57.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.017+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 zone MyZone found 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:57.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.018+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.019+0000 7fe237098800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.019+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.019+0000 7fe237098800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.019+0000 7fe237098800 20 started sync module instance, tier type = 2024-05-01T13:05:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.019+0000 7fe237098800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=0 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=1 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=2 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=3 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=4 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=5 2024-05-01T13:05:57.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.031+0000 7fe237098800 20 add_watcher() i=6 2024-05-01T13:05:57.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.033+0000 7fe237098800 20 add_watcher() i=7 2024-05-01T13:05:57.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.033+0000 7fe237098800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:57.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.035+0000 7fe0eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.035+0000 7fe237098800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:57.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.035+0000 7fe237098800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:57.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.035+0000 7fe237098800 5 note: GC not initialized 2024-05-01T13:05:57.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.035+0000 7fe0e9fe3640 20 reqs_thread_entry: start 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 20 Filter name: none 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe05b7fe640 20 reqs_thread_entry: start 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-22_topic_1 : miss 2024-05-01T13:05:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-22_topic_1 info.flags=0x0 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.036+0000 7fe237098800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-22_topic_1 to cache LRU end 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.037+0000 7fe237098800 1 failed to read topic info for name: ashubx-22_topic_1 tenant: , ret=-2 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.037+0000 7fe237098800 10 WARNING: topic name:ashubx-22_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:05:57.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.037+0000 7fe0eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.040+0000 7fe237098800 20 remove_watcher() i=0 2024-05-01T13:05:57.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.040+0000 7fe237098800 2 removed watcher, disabling cache 2024-05-01T13:05:57.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.041+0000 7fe237098800 20 remove_watcher() i=1 2024-05-01T13:05:57.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.042+0000 7fe237098800 20 remove_watcher() i=2 2024-05-01T13:05:57.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.043+0000 7fe237098800 20 remove_watcher() i=3 2024-05-01T13:05:57.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.044+0000 7fe237098800 20 remove_watcher() i=4 2024-05-01T13:05:57.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.046+0000 7fe237098800 20 remove_watcher() i=5 2024-05-01T13:05:57.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.047+0000 7fe237098800 20 remove_watcher() i=6 2024-05-01T13:05:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.048+0000 7fe237098800 20 remove_watcher() i=7 2024-05-01T13:05:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.139+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.141+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.141+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.142+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.142+0000 7f7c89333800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.142+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.142+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.142+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.143+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.143+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.143+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:05:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.143+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.144+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 20 zone MyZone found 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.145+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.146+0000 7f7c89333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.146+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.146+0000 7f7c89333800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.146+0000 7f7c89333800 20 started sync module instance, tier type = 2024-05-01T13:05:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.146+0000 7f7c89333800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=0 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=1 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=2 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=3 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=4 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=5 2024-05-01T13:05:57.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.157+0000 7f7c89333800 20 add_watcher() i=6 2024-05-01T13:05:57.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.159+0000 7f7c89333800 20 add_watcher() i=7 2024-05-01T13:05:57.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.159+0000 7f7c89333800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:57.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.160+0000 7f7b3b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7c89333800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:57.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7c89333800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7c89333800 5 note: GC not initialized 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7b3a7e4640 20 reqs_thread_entry: start 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7c89333800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7c89333800 20 Filter name: none 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.161+0000 7f7aabfff640 20 reqs_thread_entry: start 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.162+0000 7f7c89333800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.162+0000 7f7c89333800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:05:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.164+0000 7f7b3b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.166+0000 7f7c89333800 20 remove_watcher() i=0 2024-05-01T13:05:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.166+0000 7f7c89333800 2 removed watcher, disabling cache 2024-05-01T13:05:57.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.167+0000 7f7c89333800 20 remove_watcher() i=1 2024-05-01T13:05:57.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.170+0000 7f7c89333800 20 remove_watcher() i=2 2024-05-01T13:05:57.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.171+0000 7f7c89333800 20 remove_watcher() i=3 2024-05-01T13:05:57.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.172+0000 7f7c89333800 20 remove_watcher() i=4 2024-05-01T13:05:57.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.173+0000 7f7c89333800 20 remove_watcher() i=5 2024-05-01T13:05:57.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.174+0000 7f7c89333800 20 remove_watcher() i=6 2024-05-01T13:05:57.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.175+0000 7f7c89333800 20 remove_watcher() i=7 2024-05-01T13:05:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:57.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.513+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.515+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.515+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.516+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.516+0000 7f572a129800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.516+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.516+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.516+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.517+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.517+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:05:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.518+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 zone MyZone found 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.519+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.520+0000 7f572a129800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.520+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.521+0000 7f572a129800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.521+0000 7f572a129800 20 started sync module instance, tier type = 2024-05-01T13:05:57.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.521+0000 7f572a129800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=0 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=1 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=2 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=3 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=4 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.530+0000 7f572a129800 20 add_watcher() i=5 2024-05-01T13:05:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.531+0000 7f572a129800 20 add_watcher() i=6 2024-05-01T13:05:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.531+0000 7f572a129800 20 add_watcher() i=7 2024-05-01T13:05:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.531+0000 7f572a129800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:57.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.533+0000 7f55d77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:57.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 5 note: GC not initialized 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f55d67fc640 20 reqs_thread_entry: start 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 20 Filter name: none 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f5547fff640 20 reqs_thread_entry: start 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.534+0000 7f572a129800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.535+0000 7f572a129800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:05:57.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.536+0000 7f55d77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.538+0000 7f572a129800 20 remove_watcher() i=0 2024-05-01T13:05:57.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.538+0000 7f572a129800 2 removed watcher, disabling cache 2024-05-01T13:05:57.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.539+0000 7f572a129800 20 remove_watcher() i=1 2024-05-01T13:05:57.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.540+0000 7f572a129800 20 remove_watcher() i=2 2024-05-01T13:05:57.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.541+0000 7f572a129800 20 remove_watcher() i=3 2024-05-01T13:05:57.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.542+0000 7f572a129800 20 remove_watcher() i=4 2024-05-01T13:05:57.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.543+0000 7f572a129800 20 remove_watcher() i=5 2024-05-01T13:05:57.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.544+0000 7f572a129800 20 remove_watcher() i=6 2024-05-01T13:05:57.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.545+0000 7f572a129800 20 remove_watcher() i=7 2024-05-01T13:05:57.600 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:05:57.601 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:05:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.636+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.638+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.638+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.639+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.639+0000 7fa417013800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:05:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.639+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.639+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.639+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.640+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:05:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.640+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.640+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:05:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.640+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:05:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:05:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.641+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 20 zone MyZone found 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 20 period zonegroup name MyZoneGroup 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.642+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.643+0000 7fa417013800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.643+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.643+0000 7fa417013800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:05:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.643+0000 7fa417013800 20 started sync module instance, tier type = 2024-05-01T13:05:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.643+0000 7fa417013800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:05:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=0 2024-05-01T13:05:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=1 2024-05-01T13:05:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=2 2024-05-01T13:05:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=3 2024-05-01T13:05:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=4 2024-05-01T13:05:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=5 2024-05-01T13:05:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.653+0000 7fa417013800 20 add_watcher() i=6 2024-05-01T13:05:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.654+0000 7fa417013800 20 add_watcher() i=7 2024-05-01T13:05:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.654+0000 7fa417013800 2 all 8 watchers are set, enabling cache 2024-05-01T13:05:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.656+0000 7fa2c8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.656+0000 7fa417013800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.656+0000 7fa417013800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.656+0000 7fa417013800 5 note: GC not initialized 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.656+0000 7fa2437fe640 20 reqs_thread_entry: start 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.657+0000 7fa417013800 20 init_complete bucket index max shards: 11 2024-05-01T13:05:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.657+0000 7fa417013800 20 Filter name: none 2024-05-01T13:05:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.657+0000 7fa2417fa640 20 reqs_thread_entry: start 2024-05-01T13:05:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.657+0000 7fa417013800 20 rados->read ofs=0 len=0 2024-05-01T13:05:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.657+0000 7fa417013800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:05:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.659+0000 7fa2c8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:05:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.661+0000 7fa417013800 20 remove_watcher() i=0 2024-05-01T13:05:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.661+0000 7fa417013800 2 removed watcher, disabling cache 2024-05-01T13:05:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.662+0000 7fa417013800 20 remove_watcher() i=1 2024-05-01T13:05:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.664+0000 7fa417013800 20 remove_watcher() i=2 2024-05-01T13:05:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.665+0000 7fa417013800 20 remove_watcher() i=3 2024-05-01T13:05:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.666+0000 7fa417013800 20 remove_watcher() i=4 2024-05-01T13:05:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.667+0000 7fa417013800 20 remove_watcher() i=5 2024-05-01T13:05:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.667+0000 7fa417013800 20 remove_watcher() i=6 2024-05-01T13:05:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:05:57.668+0000 7fa417013800 20 remove_watcher() i=7 2024-05-01T13:06:01.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~13s 2024-05-01T13:06:02.736 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:02.737 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:02.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.777+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.779+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:02.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.779+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.780+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:02.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.780+0000 7f7b60213800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:02.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.780+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.781+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:02.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.781+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.782+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:02.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.782+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.782+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:02.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.782+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.783+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:02.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 20 zone MyZone found 2024-05-01T13:06:02.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:02.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.784+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.785+0000 7f7b60213800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.785+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.786+0000 7f7b60213800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.786+0000 7f7b60213800 20 started sync module instance, tier type = 2024-05-01T13:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.786+0000 7f7b60213800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=0 2024-05-01T13:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=1 2024-05-01T13:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=2 2024-05-01T13:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=3 2024-05-01T13:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=4 2024-05-01T13:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=5 2024-05-01T13:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.798+0000 7f7b60213800 20 add_watcher() i=6 2024-05-01T13:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.799+0000 7f7b60213800 20 add_watcher() i=7 2024-05-01T13:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.799+0000 7f7b60213800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.801+0000 7f7a11fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7b60213800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7b60213800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7b60213800 5 note: GC not initialized 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7a10fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7b60213800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f7b60213800 20 Filter name: none 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.802+0000 7f798a7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.803+0000 7f7b60213800 20 rados->read ofs=0 len=0 2024-05-01T13:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.803+0000 7f7b60213800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.805+0000 7f7a11fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.807+0000 7f7b60213800 20 remove_watcher() i=0 2024-05-01T13:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.807+0000 7f7b60213800 2 removed watcher, disabling cache 2024-05-01T13:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.808+0000 7f7b60213800 20 remove_watcher() i=1 2024-05-01T13:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.810+0000 7f7b60213800 20 remove_watcher() i=2 2024-05-01T13:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.811+0000 7f7b60213800 20 remove_watcher() i=3 2024-05-01T13:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.812+0000 7f7b60213800 20 remove_watcher() i=4 2024-05-01T13:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.813+0000 7f7b60213800 20 remove_watcher() i=5 2024-05-01T13:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.814+0000 7f7b60213800 20 remove_watcher() i=6 2024-05-01T13:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:02.815+0000 7f7b60213800 20 remove_watcher() i=7 2024-05-01T13:06:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:07.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~19s 2024-05-01T13:06:07.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.924+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.926+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.926+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.927+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.927+0000 7f584b611800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.927+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.928+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.928+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.928+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.928+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.929+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.929+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.930+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:07.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.930+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.930+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:07.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.930+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.931+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:07.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.931+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.931+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 zone MyZone found 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:07.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.932+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.933+0000 7f584b611800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.933+0000 7f584b611800 20 started sync module instance, tier type = 2024-05-01T13:06:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.933+0000 7f584b611800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=0 2024-05-01T13:06:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=1 2024-05-01T13:06:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=2 2024-05-01T13:06:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=3 2024-05-01T13:06:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=4 2024-05-01T13:06:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=5 2024-05-01T13:06:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.945+0000 7f584b611800 20 add_watcher() i=6 2024-05-01T13:06:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.946+0000 7f584b611800 20 add_watcher() i=7 2024-05-01T13:06:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.946+0000 7f584b611800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.948+0000 7f56fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.948+0000 7f584b611800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.948+0000 7f584b611800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.948+0000 7f584b611800 5 note: GC not initialized 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.948+0000 7f5677fff640 20 reqs_thread_entry: start 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.949+0000 7f584b611800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.949+0000 7f584b611800 20 Filter name: none 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.949+0000 7f5675ffb640 20 reqs_thread_entry: start 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.949+0000 7f584b611800 20 rados->read ofs=0 len=0 2024-05-01T13:06:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.949+0000 7f584b611800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.951+0000 7f56fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.953+0000 7f584b611800 20 remove_watcher() i=0 2024-05-01T13:06:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.953+0000 7f584b611800 2 removed watcher, disabling cache 2024-05-01T13:06:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.954+0000 7f584b611800 20 remove_watcher() i=1 2024-05-01T13:06:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.956+0000 7f584b611800 20 remove_watcher() i=2 2024-05-01T13:06:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.957+0000 7f584b611800 20 remove_watcher() i=3 2024-05-01T13:06:07.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.958+0000 7f584b611800 20 remove_watcher() i=4 2024-05-01T13:06:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.959+0000 7f584b611800 20 remove_watcher() i=5 2024-05-01T13:06:07.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.960+0000 7f584b611800 20 remove_watcher() i=6 2024-05-01T13:06:07.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:07.962+0000 7f584b611800 20 remove_watcher() i=7 2024-05-01T13:06:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:13.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.070+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.072+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.072+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.073+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:13.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.073+0000 7f644a33f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:13.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.073+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.074+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:13.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.074+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.074+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.074+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.075+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.075+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.076+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 20 zone MyZone found 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.077+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.078+0000 7f644a33f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.078+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.078+0000 7f644a33f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.078+0000 7f644a33f800 20 started sync module instance, tier type = 2024-05-01T13:06:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.078+0000 7f644a33f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:13.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=0 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=1 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=2 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=3 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=4 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=5 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.090+0000 7f644a33f800 20 add_watcher() i=6 2024-05-01T13:06:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.091+0000 7f644a33f800 20 add_watcher() i=7 2024-05-01T13:06:13.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.091+0000 7f644a33f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.093+0000 7f62fcfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f644a33f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f644a33f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f644a33f800 5 note: GC not initialized 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f62777fe640 20 reqs_thread_entry: start 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f644a33f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f644a33f800 20 Filter name: none 2024-05-01T13:06:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.094+0000 7f62757fa640 20 reqs_thread_entry: start 2024-05-01T13:06:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.095+0000 7f644a33f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.095+0000 7f644a33f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.097+0000 7f62fcfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:13.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.099+0000 7f644a33f800 20 remove_watcher() i=0 2024-05-01T13:06:13.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.099+0000 7f644a33f800 2 removed watcher, disabling cache 2024-05-01T13:06:13.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.100+0000 7f644a33f800 20 remove_watcher() i=1 2024-05-01T13:06:13.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.102+0000 7f644a33f800 20 remove_watcher() i=2 2024-05-01T13:06:13.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.103+0000 7f644a33f800 20 remove_watcher() i=3 2024-05-01T13:06:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.104+0000 7f644a33f800 20 remove_watcher() i=4 2024-05-01T13:06:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.105+0000 7f644a33f800 20 remove_watcher() i=5 2024-05-01T13:06:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.106+0000 7f644a33f800 20 remove_watcher() i=6 2024-05-01T13:06:13.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:13.107+0000 7f644a33f800 20 remove_watcher() i=7 2024-05-01T13:06:14.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~26s 2024-05-01T13:06:16.317 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.319 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.319 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:16] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.344 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.345 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.345 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.346 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.346 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.354 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.354 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.355 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:17.355 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:17] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.216+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.218+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.218+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.219+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.219+0000 7f27ecd2d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.219+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.219+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.219+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.220+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.220+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.222+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.222+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 zone MyZone found 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.223+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.224+0000 7f27ecd2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.224+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.225+0000 7f27ecd2d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.225+0000 7f27ecd2d800 20 started sync module instance, tier type = 2024-05-01T13:06:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.225+0000 7f27ecd2d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=0 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=1 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=2 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=3 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=4 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=5 2024-05-01T13:06:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.245+0000 7f27ecd2d800 20 add_watcher() i=6 2024-05-01T13:06:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.246+0000 7f27ecd2d800 20 add_watcher() i=7 2024-05-01T13:06:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.246+0000 7f27ecd2d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.247+0000 7f269efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.247+0000 7f27ecd2d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.247+0000 7f27ecd2d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.247+0000 7f27ecd2d800 5 note: GC not initialized 2024-05-01T13:06:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.248+0000 7f269dfe3640 20 reqs_thread_entry: start 2024-05-01T13:06:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.248+0000 7f27ecd2d800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.248+0000 7f27ecd2d800 20 Filter name: none 2024-05-01T13:06:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.248+0000 7f260f7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.248+0000 7f27ecd2d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.249+0000 7f27ecd2d800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.251+0000 7f269efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:18.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.253+0000 7f27ecd2d800 20 remove_watcher() i=0 2024-05-01T13:06:18.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.253+0000 7f27ecd2d800 2 removed watcher, disabling cache 2024-05-01T13:06:18.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.254+0000 7f27ecd2d800 20 remove_watcher() i=1 2024-05-01T13:06:18.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.255+0000 7f27ecd2d800 20 remove_watcher() i=2 2024-05-01T13:06:18.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.256+0000 7f27ecd2d800 20 remove_watcher() i=3 2024-05-01T13:06:18.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.257+0000 7f27ecd2d800 20 remove_watcher() i=4 2024-05-01T13:06:18.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.258+0000 7f27ecd2d800 20 remove_watcher() i=5 2024-05-01T13:06:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.259+0000 7f27ecd2d800 20 remove_watcher() i=6 2024-05-01T13:06:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:18.260+0000 7f27ecd2d800 20 remove_watcher() i=7 2024-05-01T13:06:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:18] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:20.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~32s 2024-05-01T13:06:23.331 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:23.331 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:23.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.373+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.375+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:23.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.375+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.376+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:23.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.376+0000 7fe93e136800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:23.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.376+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.376+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:23.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.376+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.377+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:23.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.377+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.377+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:23.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.377+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.378+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:23.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.378+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.378+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:23.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.378+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:23.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 zone MyZone found 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:23.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.379+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.380+0000 7fe93e136800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:23.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.380+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.381+0000 7fe93e136800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:23.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.381+0000 7fe93e136800 20 started sync module instance, tier type = 2024-05-01T13:06:23.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.381+0000 7fe93e136800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:23.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=0 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=1 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=2 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=3 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=4 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=5 2024-05-01T13:06:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.392+0000 7fe93e136800 20 add_watcher() i=6 2024-05-01T13:06:23.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.393+0000 7fe93e136800 20 add_watcher() i=7 2024-05-01T13:06:23.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.393+0000 7fe93e136800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:23.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe7eb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe93e136800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe93e136800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe93e136800 5 note: GC not initialized 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe7ea7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe93e136800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe93e136800 20 Filter name: none 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.396+0000 7fe75bfff640 20 reqs_thread_entry: start 2024-05-01T13:06:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.397+0000 7fe93e136800 20 rados->read ofs=0 len=0 2024-05-01T13:06:23.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.397+0000 7fe93e136800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:23.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.399+0000 7fe7eb7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:23.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.401+0000 7fe93e136800 20 remove_watcher() i=0 2024-05-01T13:06:23.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.401+0000 7fe93e136800 2 removed watcher, disabling cache 2024-05-01T13:06:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.403+0000 7fe93e136800 20 remove_watcher() i=1 2024-05-01T13:06:23.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.404+0000 7fe93e136800 20 remove_watcher() i=2 2024-05-01T13:06:23.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.405+0000 7fe93e136800 20 remove_watcher() i=3 2024-05-01T13:06:23.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.406+0000 7fe93e136800 20 remove_watcher() i=4 2024-05-01T13:06:23.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.407+0000 7fe93e136800 20 remove_watcher() i=5 2024-05-01T13:06:23.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.408+0000 7fe93e136800 20 remove_watcher() i=6 2024-05-01T13:06:23.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:23.409+0000 7fe93e136800 20 remove_watcher() i=7 2024-05-01T13:06:27.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~39s 2024-05-01T13:06:28.883 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:06:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:test data path v2 migration ... ignoring --setuser ceph since I am not root 2024-05-01T13:06:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:28.965+0000 7fa48b013800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:28.965+0000 7fa48b013800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:28.969+0000 7fa348fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:29.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.058+0000 7f43f152f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:29.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.058+0000 7f43f152f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.062+0000 7f422b7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.065+0000 7f43f152f800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:29.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.080+0000 7f43f152f800 20 updating latest_epoch from 2 -> 3 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.171+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.172+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.172+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.173+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.173+0000 7fef9ef75800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.173+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.173+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.173+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.174+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.174+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.175+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 zone MyZone found 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:3 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.176+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.177+0000 7fef9ef75800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.177+0000 7fef9ef75800 20 rados->read ofs=0 len=0 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.178+0000 7fef9ef75800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.178+0000 7fef9ef75800 20 started sync module instance, tier type = 2024-05-01T13:06:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.178+0000 7fef9ef75800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=0 2024-05-01T13:06:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=1 2024-05-01T13:06:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=2 2024-05-01T13:06:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=3 2024-05-01T13:06:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=4 2024-05-01T13:06:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=5 2024-05-01T13:06:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.189+0000 7fef9ef75800 20 add_watcher() i=6 2024-05-01T13:06:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.191+0000 7fef9ef75800 20 add_watcher() i=7 2024-05-01T13:06:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.191+0000 7fef9ef75800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.193+0000 7fee52fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.193+0000 7fef9ef75800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.193+0000 7fef9ef75800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.193+0000 7fef9ef75800 5 note: GC not initialized 2024-05-01T13:06:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.194+0000 7fee51fe3640 20 reqs_thread_entry: start 2024-05-01T13:06:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.194+0000 7fef9ef75800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.194+0000 7fef9ef75800 20 Filter name: none 2024-05-01T13:06:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.194+0000 7fedc37fe640 20 reqs_thread_entry: start 2024-05-01T13:06:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.196+0000 7fef9ef75800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.198+0000 7fef9ef75800 20 updating latest_epoch from 3 -> 4 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.205+0000 7fef9ef75800 4 Committed new epoch 4 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.207+0000 7fee52fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.210+0000 7fef9ef75800 20 remove_watcher() i=0 2024-05-01T13:06:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.210+0000 7fef9ef75800 2 removed watcher, disabling cache 2024-05-01T13:06:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.211+0000 7fef9ef75800 20 remove_watcher() i=1 2024-05-01T13:06:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.212+0000 7fef9ef75800 20 remove_watcher() i=2 2024-05-01T13:06:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.213+0000 7fef9ef75800 20 remove_watcher() i=3 2024-05-01T13:06:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.214+0000 7fef9ef75800 20 remove_watcher() i=4 2024-05-01T13:06:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.215+0000 7fef9ef75800 20 remove_watcher() i=5 2024-05-01T13:06:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.216+0000 7fef9ef75800 20 remove_watcher() i=6 2024-05-01T13:06:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.217+0000 7fef9ef75800 20 remove_watcher() i=7 2024-05-01T13:06:29.396 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:29.395+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:29.396 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:29.395+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:29.396 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:29.395+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:29.396 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:29.395+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.678 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.694 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.702 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.706 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.710 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.715 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.715 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.719 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:29] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.800+0000 7f75a292f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.800+0000 7f75a292f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.804+0000 7f7460fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.888+0000 7f9d52f46800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.888+0000 7f9d52f46800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.892+0000 7f9b96ffd640 20 reqs_thread_entry: start 2024-05-01T13:06:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.895+0000 7f9d52f46800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:29.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.907+0000 7f9d52f46800 20 updating latest_epoch from 3 -> 4 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:29.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:29.999+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.000+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:30.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.000+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.001+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.001+0000 7f4088462800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.001+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.001+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.001+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.002+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.002+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.003+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:06:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.003+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.004+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:30.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.004+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.004+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:30.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.004+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 zone MyZone found 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:4 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.005+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.006+0000 7f4088462800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.006+0000 7f4088462800 20 rados->read ofs=0 len=0 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.006+0000 7f4088462800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.006+0000 7f4088462800 20 started sync module instance, tier type = 2024-05-01T13:06:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.006+0000 7f4088462800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=0 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=1 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=2 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=3 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=4 2024-05-01T13:06:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=5 2024-05-01T13:06:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.016+0000 7f4088462800 20 add_watcher() i=6 2024-05-01T13:06:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.017+0000 7f4088462800 20 add_watcher() i=7 2024-05-01T13:06:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.017+0000 7f4088462800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.019+0000 7f3f3bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.019+0000 7f4088462800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.019+0000 7f4088462800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.019+0000 7f4088462800 5 note: GC not initialized 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.020+0000 7f3f3afe5640 20 reqs_thread_entry: start 2024-05-01T13:06:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.020+0000 7f4088462800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:30.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.020+0000 7f4088462800 20 Filter name: none 2024-05-01T13:06:30.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.020+0000 7f3f38fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:30.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.022+0000 7f4088462800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:30.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.024+0000 7f4088462800 20 updating latest_epoch from 4 -> 5 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.029+0000 7f4088462800 4 Committed new epoch 5 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.031+0000 7f3f3bfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:30.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.033+0000 7f4088462800 20 remove_watcher() i=0 2024-05-01T13:06:30.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.033+0000 7f4088462800 2 removed watcher, disabling cache 2024-05-01T13:06:30.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.034+0000 7f4088462800 20 remove_watcher() i=1 2024-05-01T13:06:30.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.035+0000 7f4088462800 20 remove_watcher() i=2 2024-05-01T13:06:30.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.036+0000 7f4088462800 20 remove_watcher() i=3 2024-05-01T13:06:30.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.037+0000 7f4088462800 20 remove_watcher() i=4 2024-05-01T13:06:30.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.038+0000 7f4088462800 20 remove_watcher() i=5 2024-05-01T13:06:30.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.039+0000 7f4088462800 20 remove_watcher() i=6 2024-05-01T13:06:30.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:30.040+0000 7f4088462800 20 remove_watcher() i=7 2024-05-01T13:06:30.277 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:30.277+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:30.278 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:30.277+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:30.278 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:30.277+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:30.278 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:30.277+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:31.102 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:31.103 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:31.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.142+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.144+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:31.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.144+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.145+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.145+0000 7fe39980f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.145+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.145+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.145+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.146+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.146+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.147+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.148+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.148+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:31.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 20 zone MyZone found 2024-05-01T13:06:31.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:31.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:31.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.149+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.150+0000 7fe39980f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.150+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.150+0000 7fe39980f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:31.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.150+0000 7fe39980f800 20 started sync module instance, tier type = 2024-05-01T13:06:31.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.150+0000 7fe39980f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=0 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=1 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=2 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=3 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=4 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=5 2024-05-01T13:06:31.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.160+0000 7fe39980f800 20 add_watcher() i=6 2024-05-01T13:06:31.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.161+0000 7fe39980f800 20 add_watcher() i=7 2024-05-01T13:06:31.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.161+0000 7fe39980f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:31.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.163+0000 7fe24afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.163+0000 7fe39980f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.163+0000 7fe39980f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.163+0000 7fe39980f800 5 note: GC not initialized 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe249fe3640 20 reqs_thread_entry: start 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 20 Filter name: none 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe1bb7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-23_topic_1 : miss 2024-05-01T13:06:31.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-23_topic_1 info.flags=0x0 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-23_topic_1 to cache LRU end 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 1 failed to read topic info for name: ashubx-23_topic_1 tenant: , ret=-2 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.164+0000 7fe39980f800 10 WARNING: topic name:ashubx-23_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.165+0000 7fe24afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:31.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.167+0000 7fe39980f800 20 remove_watcher() i=0 2024-05-01T13:06:31.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.167+0000 7fe39980f800 2 removed watcher, disabling cache 2024-05-01T13:06:31.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.168+0000 7fe39980f800 20 remove_watcher() i=1 2024-05-01T13:06:31.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.169+0000 7fe39980f800 20 remove_watcher() i=2 2024-05-01T13:06:31.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.170+0000 7fe39980f800 20 remove_watcher() i=3 2024-05-01T13:06:31.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.172+0000 7fe39980f800 20 remove_watcher() i=4 2024-05-01T13:06:31.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.173+0000 7fe39980f800 20 remove_watcher() i=5 2024-05-01T13:06:31.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.174+0000 7fe39980f800 20 remove_watcher() i=6 2024-05-01T13:06:31.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:31.175+0000 7fe39980f800 20 remove_watcher() i=7 2024-05-01T13:06:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.424 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.429 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:06:31] "POST / HTTP/1.1" 200 - 2024-05-01T13:06:31.950 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:06:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:test data path v2 large migration ... ignoring --setuser ceph since I am not root 2024-05-01T13:06:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.033+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.035+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.035+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.036+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.036+0000 7fd2da12e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.036+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.036+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.036+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.037+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.037+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.038+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.038+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.038+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.038+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.039+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.039+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.039+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.039+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 20 zone MyZone found 2024-05-01T13:06:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.040+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.041+0000 7fd2da12e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.041+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.041+0000 7fd2da12e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.041+0000 7fd2da12e800 20 started sync module instance, tier type = 2024-05-01T13:06:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.041+0000 7fd2da12e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=0 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=1 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=2 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=3 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=4 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=5 2024-05-01T13:06:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.051+0000 7fd2da12e800 20 add_watcher() i=6 2024-05-01T13:06:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.052+0000 7fd2da12e800 20 add_watcher() i=7 2024-05-01T13:06:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.052+0000 7fd2da12e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.054+0000 7fd1877fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd2da12e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd2da12e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd2da12e800 5 note: GC not initialized 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd1867fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd2da12e800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd2da12e800 20 Filter name: none 2024-05-01T13:06:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.055+0000 7fd0f7fff640 20 reqs_thread_entry: start 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got topic.:ashubx-17_topic_1 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got topic.:ashubx-14_topic 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got buckets.:ashubx-3_topic_1 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got buckets.:ashubx-17_topic_1 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got buckets.:ashubx-14_topic 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got topic.:ashubx-17_topic_2 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got buckets.:ashubx-17_topic_2 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.065+0000 7fd2da12e800 20 RGWRados::pool_iterate: got topic.:ashubx-3_topic_1 2024-05-01T13:06:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.066+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.066+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:06:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.066+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.066+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.067+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.067+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=226 2024-05-01T13:06:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.067+0000 7fd2da12e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.068+0000 7fd2da12e800 20 rados_obj.operate() r=0 bl.length=262 2024-05-01T13:06:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.071+0000 7fd1877fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.073+0000 7fd2da12e800 20 remove_watcher() i=0 2024-05-01T13:06:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.073+0000 7fd2da12e800 2 removed watcher, disabling cache 2024-05-01T13:06:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.074+0000 7fd2da12e800 20 remove_watcher() i=1 2024-05-01T13:06:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.075+0000 7fd2da12e800 20 remove_watcher() i=2 2024-05-01T13:06:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.076+0000 7fd2da12e800 20 remove_watcher() i=3 2024-05-01T13:06:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.077+0000 7fd2da12e800 20 remove_watcher() i=4 2024-05-01T13:06:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.078+0000 7fd2da12e800 20 remove_watcher() i=5 2024-05-01T13:06:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.079+0000 7fd2da12e800 20 remove_watcher() i=6 2024-05-01T13:06:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.080+0000 7fd2da12e800 20 remove_watcher() i=7 2024-05-01T13:06:32.135 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.135 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.171+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.173+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.173+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.174+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.174+0000 7f1716719800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.174+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.174+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.174+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.175+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.175+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.176+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.177+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.177+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 20 zone MyZone found 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.178+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.179+0000 7f1716719800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.179+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.179+0000 7f1716719800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.179+0000 7f1716719800 20 started sync module instance, tier type = 2024-05-01T13:06:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.179+0000 7f1716719800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=0 2024-05-01T13:06:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=1 2024-05-01T13:06:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=2 2024-05-01T13:06:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=3 2024-05-01T13:06:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=4 2024-05-01T13:06:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=5 2024-05-01T13:06:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.191+0000 7f1716719800 20 add_watcher() i=6 2024-05-01T13:06:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.192+0000 7f1716719800 20 add_watcher() i=7 2024-05-01T13:06:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.192+0000 7f1716719800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f15cb7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 5 note: GC not initialized 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f15ca7dc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 20 Filter name: none 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f15c37fe640 20 reqs_thread_entry: start 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.194+0000 7f1716719800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-14_topic : miss 2024-05-01T13:06:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-14_topic info.flags=0x15 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-14_topic to cache LRU end 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.195+0000 7f1716719800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.:ashubx-14_topic 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.196+0000 7f1716719800 10 removing MyZone.rgw.meta+topics+topic.:ashubx-14_topic from cache 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.196+0000 7f1716719800 10 distributing notification oid=MyZone.rgw.control:notify.7 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.:ashubx-14_topic, ofs0, ns] 2024-05-01T13:06:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.197+0000 7f15eb7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757335 cookie 94136423450880 notifier 7094 bl.length()=167 2024-05-01T13:06:32.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.198+0000 7f1716719800 10 distributing notification oid=MyZone.rgw.control:notify.4 cni=[op: 1, obj: MyZone.rgw.meta:topics:buckets.:ashubx-14_topic, ofs0, ns] 2024-05-01T13:06:32.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.198+0000 7f15eb7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757293 cookie 94136423429856 notifier 7094 bl.length()=169 2024-05-01T13:06:32.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.199+0000 7f1716719800 1 WARNING: failed to remove queue for persistent topic: (11) Resource temporarily unavailable 2024-05-01T13:06:32.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.200+0000 7f15cb7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.202+0000 7f1716719800 20 remove_watcher() i=0 2024-05-01T13:06:32.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.202+0000 7f1716719800 2 removed watcher, disabling cache 2024-05-01T13:06:32.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.203+0000 7f1716719800 20 remove_watcher() i=1 2024-05-01T13:06:32.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.204+0000 7f1716719800 20 remove_watcher() i=2 2024-05-01T13:06:32.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.205+0000 7f1716719800 20 remove_watcher() i=3 2024-05-01T13:06:32.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.206+0000 7f1716719800 20 remove_watcher() i=4 2024-05-01T13:06:32.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.207+0000 7f1716719800 20 remove_watcher() i=5 2024-05-01T13:06:32.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.208+0000 7f1716719800 20 remove_watcher() i=6 2024-05-01T13:06:32.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.209+0000 7f1716719800 20 remove_watcher() i=7 2024-05-01T13:06:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.303+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.304+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.304+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.305+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.305+0000 7f614bb38800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.305+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.305+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.305+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.306+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.306+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.307+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.308+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.308+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 zone MyZone found 2024-05-01T13:06:32.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.309+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.310+0000 7f614bb38800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.310+0000 7f614bb38800 20 started sync module instance, tier type = 2024-05-01T13:06:32.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.310+0000 7f614bb38800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=0 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=1 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=2 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=3 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=4 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=5 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.319+0000 7f614bb38800 20 add_watcher() i=6 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.320+0000 7f614bb38800 20 add_watcher() i=7 2024-05-01T13:06:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.320+0000 7f614bb38800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f5ffdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f614bb38800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f614bb38800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f614bb38800 5 note: GC not initialized 2024-05-01T13:06:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f5ffcfe1640 20 reqs_thread_entry: start 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f614bb38800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.322+0000 7f614bb38800 20 Filter name: none 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f5f767fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_1 : miss 2024-05-01T13:06:32.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 20 rados_obj.operate() r=0 bl.length=230 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_1 info.flags=0x15 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-17_topic_1 to cache LRU end 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.323+0000 7f614bb38800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_1 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.325+0000 7f614bb38800 10 removing MyZone.rgw.meta+topics+topic.:ashubx-17_topic_1 from cache 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.325+0000 7f614bb38800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.:ashubx-17_topic_1, ofs0, ns] 2024-05-01T13:06:32.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.325+0000 7f6026ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757336 cookie 94826031124288 notifier 7557 bl.length()=169 2024-05-01T13:06:32.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.326+0000 7f614bb38800 10 distributing notification oid=MyZone.rgw.control:notify.2 cni=[op: 1, obj: MyZone.rgw.meta:topics:buckets.:ashubx-17_topic_1, ofs0, ns] 2024-05-01T13:06:32.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.327+0000 7f60267fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757248 cookie 94826031116080 notifier 7557 bl.length()=171 2024-05-01T13:06:32.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.327+0000 7f614bb38800 1 WARNING: failed to remove queue for persistent topic: (11) Resource temporarily unavailable 2024-05-01T13:06:32.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.328+0000 7f5ffdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.330+0000 7f614bb38800 20 remove_watcher() i=0 2024-05-01T13:06:32.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.330+0000 7f614bb38800 2 removed watcher, disabling cache 2024-05-01T13:06:32.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.331+0000 7f614bb38800 20 remove_watcher() i=1 2024-05-01T13:06:32.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.332+0000 7f614bb38800 20 remove_watcher() i=2 2024-05-01T13:06:32.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.333+0000 7f614bb38800 20 remove_watcher() i=3 2024-05-01T13:06:32.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.334+0000 7f614bb38800 20 remove_watcher() i=4 2024-05-01T13:06:32.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.335+0000 7f614bb38800 20 remove_watcher() i=5 2024-05-01T13:06:32.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.336+0000 7f614bb38800 20 remove_watcher() i=6 2024-05-01T13:06:32.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.337+0000 7f614bb38800 20 remove_watcher() i=7 2024-05-01T13:06:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.429+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.430+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.430+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.431+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.431+0000 7f8f0da1f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.431+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.431+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.431+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.432+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.432+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.433+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.433+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.433+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.433+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.434+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.434+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.434+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.434+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 20 zone MyZone found 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.435+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.436+0000 7f8f0da1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.436+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.436+0000 7f8f0da1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.436+0000 7f8f0da1f800 20 started sync module instance, tier type = 2024-05-01T13:06:32.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.436+0000 7f8f0da1f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=0 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=1 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=2 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=3 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=4 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=5 2024-05-01T13:06:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=6 2024-05-01T13:06:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 20 add_watcher() i=7 2024-05-01T13:06:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.446+0000 7f8f0da1f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.448+0000 7f8dc1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.448+0000 7f8f0da1f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.448+0000 7f8f0da1f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.448+0000 7f8f0da1f800 5 note: GC not initialized 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8dc0fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 20 Filter name: none 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8d3a7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_2 : miss 2024-05-01T13:06:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 20 rados_obj.operate() r=0 bl.length=226 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_2 info.flags=0x15 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-17_topic_2 to cache LRU end 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.449+0000 7f8f0da1f800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.:ashubx-17_topic_2 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.451+0000 7f8f0da1f800 10 removing MyZone.rgw.meta+topics+topic.:ashubx-17_topic_2 from cache 2024-05-01T13:06:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.451+0000 7f8f0da1f800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.:ashubx-17_topic_2, ofs0, ns] 2024-05-01T13:06:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.451+0000 7f8deeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757337 cookie 94389231599200 notifier 7124 bl.length()=169 2024-05-01T13:06:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.452+0000 7f8f0da1f800 10 distributing notification oid=MyZone.rgw.control:notify.2 cni=[op: 1, obj: MyZone.rgw.meta:topics:buckets.:ashubx-17_topic_2, ofs0, ns] 2024-05-01T13:06:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.453+0000 7f8deeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757249 cookie 94389231587328 notifier 7124 bl.length()=171 2024-05-01T13:06:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.453+0000 7f8f0da1f800 1 WARNING: failed to remove queue for persistent topic: (11) Resource temporarily unavailable 2024-05-01T13:06:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.454+0000 7f8dc1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.456+0000 7f8f0da1f800 20 remove_watcher() i=0 2024-05-01T13:06:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.456+0000 7f8f0da1f800 2 removed watcher, disabling cache 2024-05-01T13:06:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.457+0000 7f8f0da1f800 20 remove_watcher() i=1 2024-05-01T13:06:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.458+0000 7f8f0da1f800 20 remove_watcher() i=2 2024-05-01T13:06:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.459+0000 7f8f0da1f800 20 remove_watcher() i=3 2024-05-01T13:06:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.459+0000 7f8f0da1f800 20 remove_watcher() i=4 2024-05-01T13:06:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.460+0000 7f8f0da1f800 20 remove_watcher() i=5 2024-05-01T13:06:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.461+0000 7f8f0da1f800 20 remove_watcher() i=6 2024-05-01T13:06:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.462+0000 7f8f0da1f800 20 remove_watcher() i=7 2024-05-01T13:06:32.517 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.518 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.554+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.555+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.555+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.556+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.556+0000 7f25cf21d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.556+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.556+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.556+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.557+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.557+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.558+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.558+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.558+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.558+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.558+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.559+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.559+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.559+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 zone MyZone found 2024-05-01T13:06:32.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.560+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.561+0000 7f25cf21d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.561+0000 7f25cf21d800 20 started sync module instance, tier type = 2024-05-01T13:06:32.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.561+0000 7f25cf21d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=0 2024-05-01T13:06:32.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=1 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=2 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=3 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=4 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=5 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.570+0000 7f25cf21d800 20 add_watcher() i=6 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.571+0000 7f25cf21d800 20 add_watcher() i=7 2024-05-01T13:06:32.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.571+0000 7f25cf21d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.572+0000 7f24837e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 5 note: GC not initialized 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f24827e4640 20 reqs_thread_entry: start 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 20 Filter name: none 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f247b7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:32.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-3_topic_1 : miss 2024-05-01T13:06:32.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.573+0000 7f25cf21d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.574+0000 7f25cf21d800 20 rados_obj.operate() r=0 bl.length=262 2024-05-01T13:06:32.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.574+0000 7f25cf21d800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-3_topic_1 info.flags=0x15 2024-05-01T13:06:32.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.574+0000 7f25cf21d800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-3_topic_1 to cache LRU end 2024-05-01T13:06:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.574+0000 7f25cf21d800 10 chain_cache_entry: cache_locator=MyZone.rgw.meta+topics+topic.:ashubx-3_topic_1 2024-05-01T13:06:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.575+0000 7f25cf21d800 10 removing MyZone.rgw.meta+topics+topic.:ashubx-3_topic_1 from cache 2024-05-01T13:06:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.575+0000 7f25cf21d800 10 distributing notification oid=MyZone.rgw.control:notify.6 cni=[op: 1, obj: MyZone.rgw.meta:topics:topic.:ashubx-3_topic_1, ofs0, ns] 2024-05-01T13:06:32.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.575+0000 7f24a3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757338 cookie 94551299861616 notifier 7142 bl.length()=168 2024-05-01T13:06:32.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.577+0000 7f25cf21d800 10 distributing notification oid=MyZone.rgw.control:notify.5 cni=[op: 1, obj: MyZone.rgw.meta:topics:buckets.:ashubx-3_topic_1, ofs0, ns] 2024-05-01T13:06:32.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.577+0000 7f24a37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757339 cookie 94551299863328 notifier 7142 bl.length()=170 2024-05-01T13:06:32.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.578+0000 7f24837e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.580+0000 7f25cf21d800 20 remove_watcher() i=0 2024-05-01T13:06:32.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.580+0000 7f25cf21d800 2 removed watcher, disabling cache 2024-05-01T13:06:32.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.581+0000 7f25cf21d800 20 remove_watcher() i=1 2024-05-01T13:06:32.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.582+0000 7f25cf21d800 20 remove_watcher() i=2 2024-05-01T13:06:32.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.583+0000 7f25cf21d800 20 remove_watcher() i=3 2024-05-01T13:06:32.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.584+0000 7f25cf21d800 20 remove_watcher() i=4 2024-05-01T13:06:32.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.585+0000 7f25cf21d800 20 remove_watcher() i=5 2024-05-01T13:06:32.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.586+0000 7f25cf21d800 20 remove_watcher() i=6 2024-05-01T13:06:32.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.587+0000 7f25cf21d800 20 remove_watcher() i=7 2024-05-01T13:06:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.686+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.688+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.688+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.689+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.689+0000 7f37a81af800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.689+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.689+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.689+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.690+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.690+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.690+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.690+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.691+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.691+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.691+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.691+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 zone MyZone found 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.692+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.693+0000 7f37a81af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.693+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.693+0000 7f37a81af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.693+0000 7f37a81af800 20 started sync module instance, tier type = 2024-05-01T13:06:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.693+0000 7f37a81af800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=0 2024-05-01T13:06:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=1 2024-05-01T13:06:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=2 2024-05-01T13:06:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=3 2024-05-01T13:06:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=4 2024-05-01T13:06:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=5 2024-05-01T13:06:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.703+0000 7f37a81af800 20 add_watcher() i=6 2024-05-01T13:06:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.705+0000 7f37a81af800 20 add_watcher() i=7 2024-05-01T13:06:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.705+0000 7f37a81af800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.707+0000 7f36577fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.707+0000 7f37a81af800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.707+0000 7f37a81af800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.707+0000 7f37a81af800 5 note: GC not initialized 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.707+0000 7f36567fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 Filter name: none 2024-05-01T13:06:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f35c7fff640 20 reqs_thread_entry: start 2024-05-01T13:06:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 : miss 2024-05-01T13:06:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 info.flags=0x0 2024-05-01T13:06:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 adding MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 to cache LRU end 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.5982735 : miss 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568792.5982735 info.flags=0x0 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.708+0000 7f37a81af800 10 adding MyZone.rgw.meta+users.keys+1714568792.5982735 to cache LRU end 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.709+0000 7f37a81af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.5982735 : hit (negative entry) 2024-05-01T13:06:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.709+0000 7f37a81af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.5982735 : hit (negative entry) 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.710+0000 7f37a81af800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 info.flags=0x17 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.710+0000 7f37a81af800 10 moving MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 to cache LRU end 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.710+0000 7f37a81af800 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom_1$superman1714568792.5982828, ofs0, ns] 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.710+0000 7f36897fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757293 cookie 94353876913184 notifier 7157 bl.length()=606 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.711+0000 7f36897fa640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 info.flags=0x17 2024-05-01T13:06:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.711+0000 7f36897fa640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom_1$superman1714568792.5982828 to cache LRU end 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.712+0000 7f37a81af800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568792.5982735 info.flags=0x7 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.712+0000 7f37a81af800 10 moving MyZone.rgw.meta+users.keys+1714568792.5982735 to cache LRU end 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.712+0000 7f37a81af800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568792.5982735, ofs0, ns] 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.713+0000 7f36897fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757296 cookie 94353876884368 notifier 7157 bl.length()=206 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.713+0000 7f36897fa640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568792.5982735 info.flags=0x7 2024-05-01T13:06:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.713+0000 7f36897fa640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568792.5982735 to cache LRU end 2024-05-01T13:06:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.714+0000 7f36577fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.716+0000 7f37a81af800 20 remove_watcher() i=0 2024-05-01T13:06:32.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.716+0000 7f37a81af800 2 removed watcher, disabling cache 2024-05-01T13:06:32.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.717+0000 7f37a81af800 20 remove_watcher() i=1 2024-05-01T13:06:32.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.718+0000 7f37a81af800 20 remove_watcher() i=2 2024-05-01T13:06:32.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.719+0000 7f37a81af800 20 remove_watcher() i=3 2024-05-01T13:06:32.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.720+0000 7f37a81af800 20 remove_watcher() i=4 2024-05-01T13:06:32.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.721+0000 7f37a81af800 20 remove_watcher() i=5 2024-05-01T13:06:32.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.722+0000 7f37a81af800 20 remove_watcher() i=6 2024-05-01T13:06:32.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.723+0000 7f37a81af800 20 remove_watcher() i=7 2024-05-01T13:06:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.815+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.816+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.816+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.817+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.817+0000 7f24a581f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.817+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.817+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.817+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.818+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.818+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.819+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.819+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.819+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.819+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.820+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.820+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.820+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.820+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 20 zone MyZone found 2024-05-01T13:06:32.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.821+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.822+0000 7f24a581f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.822+0000 7f24a581f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.822+0000 7f24a581f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.822+0000 7f24a581f800 20 started sync module instance, tier type = 2024-05-01T13:06:32.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.822+0000 7f24a581f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=0 2024-05-01T13:06:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=1 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=2 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=3 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=4 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=5 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.831+0000 7f24a581f800 20 add_watcher() i=6 2024-05-01T13:06:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.832+0000 7f24a581f800 20 add_watcher() i=7 2024-05-01T13:06:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.832+0000 7f24a581f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f2359fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f24a581f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f24a581f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f24a581f800 5 note: GC not initialized 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f2358fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.834+0000 7f24a581f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.835+0000 7f24a581f800 20 Filter name: none 2024-05-01T13:06:32.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.835+0000 7f22d27fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.844+0000 7f2359fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.846+0000 7f24a581f800 20 remove_watcher() i=0 2024-05-01T13:06:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.846+0000 7f24a581f800 2 removed watcher, disabling cache 2024-05-01T13:06:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.847+0000 7f24a581f800 20 remove_watcher() i=1 2024-05-01T13:06:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.848+0000 7f24a581f800 20 remove_watcher() i=2 2024-05-01T13:06:32.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.849+0000 7f24a581f800 20 remove_watcher() i=3 2024-05-01T13:06:32.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.850+0000 7f24a581f800 20 remove_watcher() i=4 2024-05-01T13:06:32.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.851+0000 7f24a581f800 20 remove_watcher() i=5 2024-05-01T13:06:32.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.852+0000 7f24a581f800 20 remove_watcher() i=6 2024-05-01T13:06:32.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.853+0000 7f24a581f800 20 remove_watcher() i=7 2024-05-01T13:06:32.916 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:32.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.953+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.955+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.955+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.955+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.955+0000 7f249d7af800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.955+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.956+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.956+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.956+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:32.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.956+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.957+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:32.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.957+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:32.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.958+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 20 zone MyZone found 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.959+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.960+0000 7f249d7af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.960+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.960+0000 7f249d7af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.960+0000 7f249d7af800 20 started sync module instance, tier type = 2024-05-01T13:06:32.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.960+0000 7f249d7af800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:32.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=0 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=1 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=2 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=3 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=4 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=5 2024-05-01T13:06:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.969+0000 7f249d7af800 20 add_watcher() i=6 2024-05-01T13:06:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.970+0000 7f249d7af800 20 add_watcher() i=7 2024-05-01T13:06:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.970+0000 7f249d7af800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.972+0000 7f2351fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.972+0000 7f249d7af800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:32.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.972+0000 7f249d7af800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:32.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f249d7af800 5 note: GC not initialized 2024-05-01T13:06:32.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f2350fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:32.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f249d7af800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f249d7af800 20 Filter name: none 2024-05-01T13:06:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f22ca7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f249d7af800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 : miss 2024-05-01T13:06:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.973+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 info.flags=0x0 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 adding MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 to cache LRU end 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.8651698 : miss 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 20 rados->read ofs=0 len=0 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568792.8651698 info.flags=0x0 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 adding MyZone.rgw.meta+users.keys+1714568792.8651698 to cache LRU end 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.8651698 : hit (negative entry) 2024-05-01T13:06:32.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.974+0000 7f249d7af800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568792.8651698 : hit (negative entry) 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f249d7af800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 info.flags=0x17 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f249d7af800 10 moving MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 to cache LRU end 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f249d7af800 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom_2$superman1714568792.8651788, ofs0, ns] 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f237effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757294 cookie 94416101074528 notifier 7653 bl.length()=604 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f237effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 info.flags=0x17 2024-05-01T13:06:32.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.976+0000 7f237effd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom_2$superman1714568792.8651788 to cache LRU end 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.977+0000 7f249d7af800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568792.8651698 info.flags=0x7 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.977+0000 7f249d7af800 10 moving MyZone.rgw.meta+users.keys+1714568792.8651698 to cache LRU end 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.977+0000 7f249d7af800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568792.8651698, ofs0, ns] 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.978+0000 7f237e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757297 cookie 94416101047056 notifier 7653 bl.length()=206 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.978+0000 7f237e7fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568792.8651698 info.flags=0x7 2024-05-01T13:06:32.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.978+0000 7f237e7fc640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568792.8651698 to cache LRU end 2024-05-01T13:06:32.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.979+0000 7f2351fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:32.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.981+0000 7f249d7af800 20 remove_watcher() i=0 2024-05-01T13:06:32.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.981+0000 7f249d7af800 2 removed watcher, disabling cache 2024-05-01T13:06:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.982+0000 7f249d7af800 20 remove_watcher() i=1 2024-05-01T13:06:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.983+0000 7f249d7af800 20 remove_watcher() i=2 2024-05-01T13:06:32.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.984+0000 7f249d7af800 20 remove_watcher() i=3 2024-05-01T13:06:32.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.985+0000 7f249d7af800 20 remove_watcher() i=4 2024-05-01T13:06:32.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.986+0000 7f249d7af800 20 remove_watcher() i=5 2024-05-01T13:06:32.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.987+0000 7f249d7af800 20 remove_watcher() i=6 2024-05-01T13:06:32.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:32.988+0000 7f249d7af800 20 remove_watcher() i=7 2024-05-01T13:06:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:33.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.080+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.082+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.082+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.083+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.083+0000 7fb99012d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.083+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.083+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.083+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.084+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.084+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:33.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.085+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.086+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.086+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:33.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 20 zone MyZone found 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:33.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.087+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.088+0000 7fb99012d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:33.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.088+0000 7fb99012d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.088+0000 7fb99012d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:33.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.088+0000 7fb99012d800 20 started sync module instance, tier type = 2024-05-01T13:06:33.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.088+0000 7fb99012d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=0 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=1 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=2 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=3 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=4 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=5 2024-05-01T13:06:33.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.097+0000 7fb99012d800 20 add_watcher() i=6 2024-05-01T13:06:33.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.098+0000 7fb99012d800 20 add_watcher() i=7 2024-05-01T13:06:33.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.098+0000 7fb99012d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:33.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.099+0000 7fb841fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:33.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb99012d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:33.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb99012d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:33.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb99012d800 5 note: GC not initialized 2024-05-01T13:06:33.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb840fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:33.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb99012d800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:33.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb99012d800 20 Filter name: none 2024-05-01T13:06:33.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.100+0000 7fb7ba7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.111+0000 7fb841fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:33.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.113+0000 7fb99012d800 20 remove_watcher() i=0 2024-05-01T13:06:33.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.113+0000 7fb99012d800 2 removed watcher, disabling cache 2024-05-01T13:06:33.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.114+0000 7fb99012d800 20 remove_watcher() i=1 2024-05-01T13:06:33.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.115+0000 7fb99012d800 20 remove_watcher() i=2 2024-05-01T13:06:33.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.116+0000 7fb99012d800 20 remove_watcher() i=3 2024-05-01T13:06:33.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.117+0000 7fb99012d800 20 remove_watcher() i=4 2024-05-01T13:06:33.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.118+0000 7fb99012d800 20 remove_watcher() i=5 2024-05-01T13:06:33.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.119+0000 7fb99012d800 20 remove_watcher() i=6 2024-05-01T13:06:33.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.120+0000 7fb99012d800 20 remove_watcher() i=7 2024-05-01T13:06:33.174 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:33.174 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:33.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.201+0000 7fecd0323800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:33.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.201+0000 7fecd0323800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:33.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.205+0000 7feb8a7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:33.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.293+0000 7f2789887800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:33.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.293+0000 7f2789887800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:33.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.297+0000 7f25cdffb640 20 reqs_thread_entry: start 2024-05-01T13:06:33.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.300+0000 7f2789887800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.313+0000 7f2789887800 20 updating latest_epoch from 4 -> 5 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:33.369 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:33.369 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:33.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.406+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.407+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.407+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.407+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.407+0000 7f73e331f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:33.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.407+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.408+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.408+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.409+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:33.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.409+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.409+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.409+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.410+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.410+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.410+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.410+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:33.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 zone MyZone found 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:5 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:33.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.411+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.412+0000 7f73e331f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:33.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.412+0000 7f73e331f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:33.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.412+0000 7f73e331f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:33.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.412+0000 7f73e331f800 20 started sync module instance, tier type = 2024-05-01T13:06:33.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.412+0000 7f73e331f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=0 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=1 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=2 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=3 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=4 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=5 2024-05-01T13:06:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.421+0000 7f73e331f800 20 add_watcher() i=6 2024-05-01T13:06:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.423+0000 7f73e331f800 20 add_watcher() i=7 2024-05-01T13:06:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.423+0000 7f73e331f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:33.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.424+0000 7f72957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:33.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f73e331f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f73e331f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f73e331f800 5 note: GC not initialized 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f720ffff640 20 reqs_thread_entry: start 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f73e331f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f73e331f800 20 Filter name: none 2024-05-01T13:06:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.425+0000 7f720dffb640 20 reqs_thread_entry: start 2024-05-01T13:06:33.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.427+0000 7f73e331f800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:33.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.429+0000 7f73e331f800 20 updating latest_epoch from 5 -> 6 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:33.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.438+0000 7f73e331f800 4 Committed new epoch 6 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:33.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.440+0000 7f72957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:33.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.442+0000 7f73e331f800 20 remove_watcher() i=0 2024-05-01T13:06:33.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.442+0000 7f73e331f800 2 removed watcher, disabling cache 2024-05-01T13:06:33.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.443+0000 7f73e331f800 20 remove_watcher() i=1 2024-05-01T13:06:33.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.444+0000 7f73e331f800 20 remove_watcher() i=2 2024-05-01T13:06:33.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.445+0000 7f73e331f800 20 remove_watcher() i=3 2024-05-01T13:06:33.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.446+0000 7f73e331f800 20 remove_watcher() i=4 2024-05-01T13:06:33.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.447+0000 7f73e331f800 20 remove_watcher() i=5 2024-05-01T13:06:33.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.448+0000 7f73e331f800 20 remove_watcher() i=6 2024-05-01T13:06:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:33.449+0000 7f73e331f800 20 remove_watcher() i=7 2024-05-01T13:06:33.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~45s 2024-05-01T13:06:33.560 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:33.560+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:33.560 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:33.560+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:33.560 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:33.560+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:33.560 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:33.560+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:35.144 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:35.144 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:35.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.172+0000 7f37cb725800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:35.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.172+0000 7f37cb725800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:35.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.176+0000 7f360dffb640 20 reqs_thread_entry: start 2024-05-01T13:06:35.232 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:35.232 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.259+0000 7fa110f33800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.259+0000 7fa110f33800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.262+0000 7f9f52ffd640 20 reqs_thread_entry: start 2024-05-01T13:06:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.266+0000 7fa110f33800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.279+0000 7fa110f33800 20 updating latest_epoch from 5 -> 6 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:35.335 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:35.335 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:35.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.371+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.372+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.372+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.373+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.373+0000 7f1a7a527800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:35.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.373+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.373+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.373+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.374+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.374+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.375+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.375+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.375+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.375+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.376+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.376+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.376+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.376+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 zone MyZone found 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:6 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.377+0000 7f1a7a527800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.378+0000 7f1a7a527800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.378+0000 7f1a7a527800 20 started sync module instance, tier type = 2024-05-01T13:06:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.378+0000 7f1a7a527800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=0 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=1 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=2 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=3 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=4 2024-05-01T13:06:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.388+0000 7f1a7a527800 20 add_watcher() i=5 2024-05-01T13:06:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.389+0000 7f1a7a527800 20 add_watcher() i=6 2024-05-01T13:06:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.389+0000 7f1a7a527800 20 add_watcher() i=7 2024-05-01T13:06:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.389+0000 7f1a7a527800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.391+0000 7f192cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.391+0000 7f1a7a527800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.391+0000 7f1a7a527800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.391+0000 7f1a7a527800 5 note: GC not initialized 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.391+0000 7f18a77fe640 20 reqs_thread_entry: start 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.392+0000 7f1a7a527800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.392+0000 7f1a7a527800 20 Filter name: none 2024-05-01T13:06:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.392+0000 7f18a57fa640 20 reqs_thread_entry: start 2024-05-01T13:06:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.393+0000 7f1a7a527800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.395+0000 7f1a7a527800 20 updating latest_epoch from 6 -> 7 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.402+0000 7f1a7a527800 4 Committed new epoch 7 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.404+0000 7f192cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.406+0000 7f1a7a527800 20 remove_watcher() i=0 2024-05-01T13:06:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.406+0000 7f1a7a527800 2 removed watcher, disabling cache 2024-05-01T13:06:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.407+0000 7f1a7a527800 20 remove_watcher() i=1 2024-05-01T13:06:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.408+0000 7f1a7a527800 20 remove_watcher() i=2 2024-05-01T13:06:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.409+0000 7f1a7a527800 20 remove_watcher() i=3 2024-05-01T13:06:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.410+0000 7f1a7a527800 20 remove_watcher() i=4 2024-05-01T13:06:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.411+0000 7f1a7a527800 20 remove_watcher() i=5 2024-05-01T13:06:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.412+0000 7f1a7a527800 20 remove_watcher() i=6 2024-05-01T13:06:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.413+0000 7f1a7a527800 20 remove_watcher() i=7 2024-05-01T13:06:35.468 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:35.469 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:35.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.504+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.506+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.506+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.507+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.507+0000 7f87ae20f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.507+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.507+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.507+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.508+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.508+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.508+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.508+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.509+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.509+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.509+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.509+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 zone MyZone found 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.510+0000 7f87ae20f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 started sync module instance, tier type = 2024-05-01T13:06:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.511+0000 7f87ae20f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=0 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=1 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=2 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=3 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=4 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=5 2024-05-01T13:06:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.520+0000 7f87ae20f800 20 add_watcher() i=6 2024-05-01T13:06:35.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.521+0000 7f87ae20f800 20 add_watcher() i=7 2024-05-01T13:06:35.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.521+0000 7f87ae20f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f865b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f87ae20f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f87ae20f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f87ae20f800 5 note: GC not initialized 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f865a7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.523+0000 7f87ae20f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f87ae20f800 20 Filter name: none 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f85cbfff640 20 reqs_thread_entry: start 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f87ae20f800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f87ae20f800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x6 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:ERROR: could not remove topic: 2024-05-01T13:06:35.524+0000 7f87ae20f800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:(2202) Unknown error 2202 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f87ae20f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T13:06:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.524+0000 7f87ae20f800 1 WARNING: topic migration in process. please try again later 2024-05-01T13:06:35.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.525+0000 7f865b7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:35.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.527+0000 7f87ae20f800 20 remove_watcher() i=0 2024-05-01T13:06:35.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.527+0000 7f87ae20f800 2 removed watcher, disabling cache 2024-05-01T13:06:35.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.528+0000 7f87ae20f800 20 remove_watcher() i=1 2024-05-01T13:06:35.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.529+0000 7f87ae20f800 20 remove_watcher() i=2 2024-05-01T13:06:35.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.530+0000 7f87ae20f800 20 remove_watcher() i=3 2024-05-01T13:06:35.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.531+0000 7f87ae20f800 20 remove_watcher() i=4 2024-05-01T13:06:35.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.532+0000 7f87ae20f800 20 remove_watcher() i=5 2024-05-01T13:06:35.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.533+0000 7f87ae20f800 20 remove_watcher() i=6 2024-05-01T13:06:35.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:35.534+0000 7f87ae20f800 20 remove_watcher() i=7 2024-05-01T13:06:35.638 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:35.637+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:35.638 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:35.638+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:35.639 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:35.638+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:35.639 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:35.638+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:36.595 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:36.595 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:36.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.634+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.636+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.636+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.636+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.636+0000 7f84dc011800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.636+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.637+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.637+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.638+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.638+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.638+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.638+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.639+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.639+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.639+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.639+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.640+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.640+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 zone MyZone found 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.641+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.642+0000 7f84dc011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.642+0000 7f84dc011800 20 started sync module instance, tier type = 2024-05-01T13:06:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.642+0000 7f84dc011800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=0 2024-05-01T13:06:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=1 2024-05-01T13:06:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=2 2024-05-01T13:06:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=3 2024-05-01T13:06:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=4 2024-05-01T13:06:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=5 2024-05-01T13:06:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.653+0000 7f84dc011800 20 add_watcher() i=6 2024-05-01T13:06:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.654+0000 7f84dc011800 20 add_watcher() i=7 2024-05-01T13:06:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.654+0000 7f84dc011800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f838dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f84dc011800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f84dc011800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f84dc011800 5 note: GC not initialized 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f838cfe1640 20 reqs_thread_entry: start 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f84dc011800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.656+0000 7f84dc011800 20 Filter name: none 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f83067fc640 20 reqs_thread_entry: start 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-24_topic_1 : miss 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-24_topic_1 info.flags=0x0 2024-05-01T13:06:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-24_topic_1 to cache LRU end 2024-05-01T13:06:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 1 failed to read topic info for name: ashubx-24_topic_1 tenant: , ret=-2 2024-05-01T13:06:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.657+0000 7f84dc011800 10 WARNING: topic name:ashubx-24_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.658+0000 7f838dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.660+0000 7f84dc011800 20 remove_watcher() i=0 2024-05-01T13:06:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.660+0000 7f84dc011800 2 removed watcher, disabling cache 2024-05-01T13:06:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.661+0000 7f84dc011800 20 remove_watcher() i=1 2024-05-01T13:06:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.663+0000 7f84dc011800 20 remove_watcher() i=2 2024-05-01T13:06:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.664+0000 7f84dc011800 20 remove_watcher() i=3 2024-05-01T13:06:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.665+0000 7f84dc011800 20 remove_watcher() i=4 2024-05-01T13:06:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.666+0000 7f84dc011800 20 remove_watcher() i=5 2024-05-01T13:06:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.667+0000 7f84dc011800 20 remove_watcher() i=6 2024-05-01T13:06:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.668+0000 7f84dc011800 20 remove_watcher() i=7 2024-05-01T13:06:36.725 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:36.725 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:36.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.761+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.763+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.763+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.764+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.764+0000 7f664df23800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.764+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.764+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.764+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.765+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.765+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.766+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.767+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.767+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 zone MyZone found 2024-05-01T13:06:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.768+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.769+0000 7f664df23800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.769+0000 7f664df23800 20 started sync module instance, tier type = 2024-05-01T13:06:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.769+0000 7f664df23800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=0 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=1 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=2 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=3 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=4 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=5 2024-05-01T13:06:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.779+0000 7f664df23800 20 add_watcher() i=6 2024-05-01T13:06:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.780+0000 7f664df23800 20 add_watcher() i=7 2024-05-01T13:06:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.780+0000 7f664df23800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.781+0000 7f64f37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 5 note: GC not initialized 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f64f27fc640 20 reqs_thread_entry: start 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 20 Filter name: none 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f646bfff640 20 reqs_thread_entry: start 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_1 : miss 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.782+0000 7f664df23800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom_1 info.flags=0x0 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 adding MyZone.rgw.log++pubsub.kaboom_1 to cache LRU end 2024-05-01T13:06:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_1 : hit (negative entry) 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-25_topic_1 : miss 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-25_topic_1 info.flags=0x0 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 adding MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-25_topic_1 to cache LRU end 2024-05-01T13:06:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 1 failed to read topic info for name: ashubx-25_topic_1 tenant: kaboom_1, ret=-2 2024-05-01T13:06:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.783+0000 7f664df23800 10 WARNING: topic name:ashubx-25_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.784+0000 7f64f37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.785+0000 7f664df23800 20 remove_watcher() i=0 2024-05-01T13:06:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.785+0000 7f664df23800 2 removed watcher, disabling cache 2024-05-01T13:06:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.786+0000 7f664df23800 20 remove_watcher() i=1 2024-05-01T13:06:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.788+0000 7f664df23800 20 remove_watcher() i=2 2024-05-01T13:06:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.789+0000 7f664df23800 20 remove_watcher() i=3 2024-05-01T13:06:36.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.790+0000 7f664df23800 20 remove_watcher() i=4 2024-05-01T13:06:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.791+0000 7f664df23800 20 remove_watcher() i=5 2024-05-01T13:06:36.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.792+0000 7f664df23800 20 remove_watcher() i=6 2024-05-01T13:06:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.793+0000 7f664df23800 20 remove_watcher() i=7 2024-05-01T13:06:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.886+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.887+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.887+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.888+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.888+0000 7f71aad19800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.888+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.888+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.888+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.889+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.889+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.889+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.889+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.890+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.890+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.890+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.890+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.891+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.891+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 zone MyZone found 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.892+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.893+0000 7f71aad19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.893+0000 7f71aad19800 20 started sync module instance, tier type = 2024-05-01T13:06:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.893+0000 7f71aad19800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=0 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=1 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=2 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=3 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=4 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=5 2024-05-01T13:06:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.902+0000 7f71aad19800 20 add_watcher() i=6 2024-05-01T13:06:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.903+0000 7f71aad19800 20 add_watcher() i=7 2024-05-01T13:06:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.903+0000 7f71aad19800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.905+0000 7f705f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.905+0000 7f71aad19800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.905+0000 7f71aad19800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.905+0000 7f71aad19800 5 note: GC not initialized 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f705e7e4640 20 reqs_thread_entry: start 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 20 Filter name: none 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f7057fff640 20 reqs_thread_entry: start 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_2 : miss 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom_2 info.flags=0x0 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 10 adding MyZone.rgw.log++pubsub.kaboom_2 to cache LRU end 2024-05-01T13:06:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_2 : hit (negative entry) 2024-05-01T13:06:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-26_topic_1 : miss 2024-05-01T13:06:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.906+0000 7f71aad19800 20 rados->read ofs=0 len=0 2024-05-01T13:06:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.907+0000 7f71aad19800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.907+0000 7f71aad19800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-26_topic_1 info.flags=0x0 2024-05-01T13:06:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.907+0000 7f71aad19800 10 adding MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-26_topic_1 to cache LRU end 2024-05-01T13:06:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.907+0000 7f71aad19800 1 failed to read topic info for name: ashubx-26_topic_1 tenant: kaboom_2, ret=-2 2024-05-01T13:06:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.907+0000 7f71aad19800 10 WARNING: topic name:ashubx-26_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.908+0000 7f705f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.910+0000 7f71aad19800 20 remove_watcher() i=0 2024-05-01T13:06:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.910+0000 7f71aad19800 2 removed watcher, disabling cache 2024-05-01T13:06:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.911+0000 7f71aad19800 20 remove_watcher() i=1 2024-05-01T13:06:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.912+0000 7f71aad19800 20 remove_watcher() i=2 2024-05-01T13:06:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.913+0000 7f71aad19800 20 remove_watcher() i=3 2024-05-01T13:06:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.914+0000 7f71aad19800 20 remove_watcher() i=4 2024-05-01T13:06:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.915+0000 7f71aad19800 20 remove_watcher() i=5 2024-05-01T13:06:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.916+0000 7f71aad19800 20 remove_watcher() i=6 2024-05-01T13:06:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:36.917+0000 7f71aad19800 20 remove_watcher() i=7 2024-05-01T13:06:36.970 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:36.970 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.006+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.008+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.008+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.009+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.009+0000 7fdd9821f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.009+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.009+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.009+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.010+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.010+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.010+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.011+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.011+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.011+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.011+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.011+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 zone MyZone found 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.012+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.013+0000 7fdd9821f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.013+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.013+0000 7fdd9821f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.013+0000 7fdd9821f800 20 started sync module instance, tier type = 2024-05-01T13:06:37.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.013+0000 7fdd9821f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=0 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=1 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=2 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=3 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=4 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=5 2024-05-01T13:06:37.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.022+0000 7fdd9821f800 20 add_watcher() i=6 2024-05-01T13:06:37.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.023+0000 7fdd9821f800 20 add_watcher() i=7 2024-05-01T13:06:37.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.023+0000 7fdd9821f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.025+0000 7fdc477fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdd9821f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdd9821f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdd9821f800 5 note: GC not initialized 2024-05-01T13:06:37.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdc467fc640 20 reqs_thread_entry: start 2024-05-01T13:06:37.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdd9821f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdd9821f800 20 Filter name: none 2024-05-01T13:06:37.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.026+0000 7fdbb7fff640 20 reqs_thread_entry: start 2024-05-01T13:06:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.036+0000 7fdd9821f800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-26_topic 2024-05-01T13:06:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.036+0000 7fdd9821f800 20 RGWRados::pool_iterate: got topic.:ashubx-24_topic 2024-05-01T13:06:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.036+0000 7fdd9821f800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-25_topic 2024-05-01T13:06:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.036+0000 7fdd9821f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.036+0000 7fdd9821f800 20 rados_obj.operate() r=0 bl.length=222 2024-05-01T13:06:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.038+0000 7fdc477fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.040+0000 7fdd9821f800 20 remove_watcher() i=0 2024-05-01T13:06:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.040+0000 7fdd9821f800 2 removed watcher, disabling cache 2024-05-01T13:06:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.041+0000 7fdd9821f800 20 remove_watcher() i=1 2024-05-01T13:06:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.042+0000 7fdd9821f800 20 remove_watcher() i=2 2024-05-01T13:06:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.043+0000 7fdd9821f800 20 remove_watcher() i=3 2024-05-01T13:06:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.044+0000 7fdd9821f800 20 remove_watcher() i=4 2024-05-01T13:06:37.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.045+0000 7fdd9821f800 20 remove_watcher() i=5 2024-05-01T13:06:37.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.046+0000 7fdd9821f800 20 remove_watcher() i=6 2024-05-01T13:06:37.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.047+0000 7fdd9821f800 20 remove_watcher() i=7 2024-05-01T13:06:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.139+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.140+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.140+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.141+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.141+0000 7ff6ae011800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.141+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.141+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.141+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.142+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.142+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.143+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.143+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.143+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.143+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.143+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.144+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.144+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.144+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 zone MyZone found 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.145+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.146+0000 7ff6ae011800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.146+0000 7ff6ae011800 20 started sync module instance, tier type = 2024-05-01T13:06:37.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.146+0000 7ff6ae011800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=0 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=1 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=2 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=3 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=4 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=5 2024-05-01T13:06:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=6 2024-05-01T13:06:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 20 add_watcher() i=7 2024-05-01T13:06:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.157+0000 7ff6ae011800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.159+0000 7ff55f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff6ae011800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff6ae011800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff6ae011800 5 note: GC not initialized 2024-05-01T13:06:37.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff55e7e4640 20 reqs_thread_entry: start 2024-05-01T13:06:37.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff6ae011800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff6ae011800 20 Filter name: none 2024-05-01T13:06:37.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.160+0000 7ff4cffff640 20 reqs_thread_entry: start 2024-05-01T13:06:37.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.170+0000 7ff6ae011800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-26_topic 2024-05-01T13:06:37.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.170+0000 7ff6ae011800 20 RGWRados::pool_iterate: got topic.:ashubx-24_topic 2024-05-01T13:06:37.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.170+0000 7ff6ae011800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-25_topic 2024-05-01T13:06:37.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.170+0000 7ff6ae011800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.170+0000 7ff6ae011800 20 rados_obj.operate() r=0 bl.length=260 2024-05-01T13:06:37.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.172+0000 7ff55f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.174+0000 7ff6ae011800 20 remove_watcher() i=0 2024-05-01T13:06:37.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.174+0000 7ff6ae011800 2 removed watcher, disabling cache 2024-05-01T13:06:37.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.175+0000 7ff6ae011800 20 remove_watcher() i=1 2024-05-01T13:06:37.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.176+0000 7ff6ae011800 20 remove_watcher() i=2 2024-05-01T13:06:37.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.177+0000 7ff6ae011800 20 remove_watcher() i=3 2024-05-01T13:06:37.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.178+0000 7ff6ae011800 20 remove_watcher() i=4 2024-05-01T13:06:37.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.179+0000 7ff6ae011800 20 remove_watcher() i=5 2024-05-01T13:06:37.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.180+0000 7ff6ae011800 20 remove_watcher() i=6 2024-05-01T13:06:37.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.181+0000 7ff6ae011800 20 remove_watcher() i=7 2024-05-01T13:06:37.237 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:37.237 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.273+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.274+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.274+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.275+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.275+0000 7f4df4740800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.275+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.275+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.275+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.276+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.276+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.277+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.278+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.278+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 zone MyZone found 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 started sync module instance, tier type = 2024-05-01T13:06:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.279+0000 7f4df4740800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=0 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=1 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=2 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=3 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=4 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=5 2024-05-01T13:06:37.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.289+0000 7f4df4740800 20 add_watcher() i=6 2024-05-01T13:06:37.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.290+0000 7f4df4740800 20 add_watcher() i=7 2024-05-01T13:06:37.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.290+0000 7f4df4740800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.292+0000 7f4ca8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4df4740800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4df4740800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4df4740800 5 note: GC not initialized 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4c237fe640 20 reqs_thread_entry: start 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4df4740800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4df4740800 20 Filter name: none 2024-05-01T13:06:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.293+0000 7f4c217fa640 20 reqs_thread_entry: start 2024-05-01T13:06:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.303+0000 7f4df4740800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-26_topic 2024-05-01T13:06:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.303+0000 7f4df4740800 20 RGWRados::pool_iterate: got topic.:ashubx-24_topic 2024-05-01T13:06:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.303+0000 7f4df4740800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-25_topic 2024-05-01T13:06:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.303+0000 7f4df4740800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.303+0000 7f4df4740800 20 rados_obj.operate() r=0 bl.length=260 2024-05-01T13:06:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.305+0000 7f4ca8fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.307+0000 7f4df4740800 20 remove_watcher() i=0 2024-05-01T13:06:37.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.307+0000 7f4df4740800 2 removed watcher, disabling cache 2024-05-01T13:06:37.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.308+0000 7f4df4740800 20 remove_watcher() i=1 2024-05-01T13:06:37.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.309+0000 7f4df4740800 20 remove_watcher() i=2 2024-05-01T13:06:37.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.310+0000 7f4df4740800 20 remove_watcher() i=3 2024-05-01T13:06:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.311+0000 7f4df4740800 20 remove_watcher() i=4 2024-05-01T13:06:37.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.312+0000 7f4df4740800 20 remove_watcher() i=5 2024-05-01T13:06:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.313+0000 7f4df4740800 20 remove_watcher() i=6 2024-05-01T13:06:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.315+0000 7f4df4740800 20 remove_watcher() i=7 2024-05-01T13:06:37.369 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:37.369 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.406+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.407+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.407+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.408+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.408+0000 7fd655a62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.408+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.408+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.408+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.409+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.409+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.410+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.410+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.410+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.410+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.411+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.411+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.411+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.411+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 zone MyZone found 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.412+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.413+0000 7fd655a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.413+0000 7fd655a62800 20 started sync module instance, tier type = 2024-05-01T13:06:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.413+0000 7fd655a62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=0 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=1 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=2 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=3 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=4 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=5 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.422+0000 7fd655a62800 20 add_watcher() i=6 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.423+0000 7fd655a62800 20 add_watcher() i=7 2024-05-01T13:06:37.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.423+0000 7fd655a62800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.425+0000 7fd509fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.425+0000 7fd655a62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.425+0000 7fd655a62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.425+0000 7fd655a62800 5 note: GC not initialized 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd508fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd655a62800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd655a62800 20 Filter name: none 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd4827fc640 20 reqs_thread_entry: start 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=187 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.426+0000 7fd655a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.427+0000 7fd655a62800 20 rados_obj.operate() r=0 bl.length=401 2024-05-01T13:06:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.427+0000 7fd655a62800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.429+0000 7fd509fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.431+0000 7fd655a62800 20 remove_watcher() i=0 2024-05-01T13:06:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.431+0000 7fd655a62800 2 removed watcher, disabling cache 2024-05-01T13:06:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.432+0000 7fd655a62800 20 remove_watcher() i=1 2024-05-01T13:06:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.433+0000 7fd655a62800 20 remove_watcher() i=2 2024-05-01T13:06:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.434+0000 7fd655a62800 20 remove_watcher() i=3 2024-05-01T13:06:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.435+0000 7fd655a62800 20 remove_watcher() i=4 2024-05-01T13:06:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.435+0000 7fd655a62800 20 remove_watcher() i=5 2024-05-01T13:06:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.436+0000 7fd655a62800 20 remove_watcher() i=6 2024-05-01T13:06:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.438+0000 7fd655a62800 20 remove_watcher() i=7 2024-05-01T13:06:37.495 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:37.495 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.531+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.533+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.533+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.533+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.533+0000 7f9fb1346800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.533+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.534+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.534+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.535+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.535+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.535+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.535+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.536+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.536+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.536+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.536+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 zone MyZone found 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.537+0000 7f9fb1346800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 started sync module instance, tier type = 2024-05-01T13:06:37.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.538+0000 7f9fb1346800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=0 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=1 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=2 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=3 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=4 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=5 2024-05-01T13:06:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.548+0000 7f9fb1346800 20 add_watcher() i=6 2024-05-01T13:06:37.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.549+0000 7f9fb1346800 20 add_watcher() i=7 2024-05-01T13:06:37.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.549+0000 7f9fb1346800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.550+0000 7f9e657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.550+0000 7f9fb1346800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.550+0000 7f9fb1346800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.550+0000 7f9fb1346800 5 note: GC not initialized 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9ddffff640 20 reqs_thread_entry: start 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9fb1346800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9fb1346800 20 Filter name: none 2024-05-01T13:06:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9dddffb640 20 reqs_thread_entry: start 2024-05-01T13:06:37.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=231 2024-05-01T13:06:37.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.551+0000 7f9fb1346800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.552+0000 7f9fb1346800 20 rados_obj.operate() r=0 bl.length=453 2024-05-01T13:06:37.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.552+0000 7f9fb1346800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:37.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.554+0000 7f9e657e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.556+0000 7f9fb1346800 20 remove_watcher() i=0 2024-05-01T13:06:37.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.556+0000 7f9fb1346800 2 removed watcher, disabling cache 2024-05-01T13:06:37.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.557+0000 7f9fb1346800 20 remove_watcher() i=1 2024-05-01T13:06:37.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.558+0000 7f9fb1346800 20 remove_watcher() i=2 2024-05-01T13:06:37.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.559+0000 7f9fb1346800 20 remove_watcher() i=3 2024-05-01T13:06:37.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.560+0000 7f9fb1346800 20 remove_watcher() i=4 2024-05-01T13:06:37.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.561+0000 7f9fb1346800 20 remove_watcher() i=5 2024-05-01T13:06:37.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.562+0000 7f9fb1346800 20 remove_watcher() i=6 2024-05-01T13:06:37.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.563+0000 7f9fb1346800 20 remove_watcher() i=7 2024-05-01T13:06:37.620 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:37.621 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.656+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.658+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.658+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.659+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.659+0000 7f7b9b411800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.659+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.659+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.659+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.660+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.660+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.661+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.661+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.661+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.661+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.662+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.662+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.662+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.662+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 20 zone MyZone found 2024-05-01T13:06:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.663+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.664+0000 7f7b9b411800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.664+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.664+0000 7f7b9b411800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.664+0000 7f7b9b411800 20 started sync module instance, tier type = 2024-05-01T13:06:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.664+0000 7f7b9b411800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=0 2024-05-01T13:06:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=1 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=2 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=3 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=4 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=5 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.673+0000 7f7b9b411800 20 add_watcher() i=6 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.674+0000 7f7b9b411800 20 add_watcher() i=7 2024-05-01T13:06:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.674+0000 7f7b9b411800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7a4d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7b9b411800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7b9b411800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7b9b411800 5 note: GC not initialized 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f79c7fff640 20 reqs_thread_entry: start 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7b9b411800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.676+0000 7f7b9b411800 20 Filter name: none 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.677+0000 7f79c5ffb640 20 reqs_thread_entry: start 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.677+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.677+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=231 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.677+0000 7f7b9b411800 20 rados->read ofs=0 len=0 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.678+0000 7f7b9b411800 20 rados_obj.operate() r=0 bl.length=453 2024-05-01T13:06:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.678+0000 7f7b9b411800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.679+0000 7f7a4d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.681+0000 7f7b9b411800 20 remove_watcher() i=0 2024-05-01T13:06:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.681+0000 7f7b9b411800 2 removed watcher, disabling cache 2024-05-01T13:06:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.682+0000 7f7b9b411800 20 remove_watcher() i=1 2024-05-01T13:06:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.683+0000 7f7b9b411800 20 remove_watcher() i=2 2024-05-01T13:06:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.684+0000 7f7b9b411800 20 remove_watcher() i=3 2024-05-01T13:06:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.686+0000 7f7b9b411800 20 remove_watcher() i=4 2024-05-01T13:06:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.686+0000 7f7b9b411800 20 remove_watcher() i=5 2024-05-01T13:06:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.687+0000 7f7b9b411800 20 remove_watcher() i=6 2024-05-01T13:06:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:37.689+0000 7f7b9b411800 20 remove_watcher() i=7 2024-05-01T13:06:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:06:38.205 INFO:teuthology.orchestra.run.smithi050.stderr:test data path v2 mixed migration ... ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.205 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.244+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.245+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.245+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.246+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.246+0000 7f92f2898800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:38.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.246+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.247+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.247+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.247+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:38.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.247+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.248+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:38.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.248+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.249+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.249+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.249+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.249+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:38.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 zone MyZone found 2024-05-01T13:06:38.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.250+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.251+0000 7f92f2898800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.251+0000 7f92f2898800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.251+0000 7f92f2898800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.252+0000 7f92f2898800 20 started sync module instance, tier type = 2024-05-01T13:06:38.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.252+0000 7f92f2898800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:38.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=0 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=1 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=2 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=3 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=4 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=5 2024-05-01T13:06:38.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.261+0000 7f92f2898800 20 add_watcher() i=6 2024-05-01T13:06:38.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.263+0000 7f92f2898800 20 add_watcher() i=7 2024-05-01T13:06:38.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.263+0000 7f92f2898800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:38.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f91a6fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f92f2898800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f92f2898800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f92f2898800 5 note: GC not initialized 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f91a5fdb640 20 reqs_thread_entry: start 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f92f2898800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f92f2898800 20 Filter name: none 2024-05-01T13:06:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.266+0000 7f91177fe640 20 reqs_thread_entry: start 2024-05-01T13:06:38.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.276+0000 7f91a6fdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.279+0000 7f92f2898800 20 remove_watcher() i=0 2024-05-01T13:06:38.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.279+0000 7f92f2898800 2 removed watcher, disabling cache 2024-05-01T13:06:38.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.280+0000 7f92f2898800 20 remove_watcher() i=1 2024-05-01T13:06:38.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.281+0000 7f92f2898800 20 remove_watcher() i=2 2024-05-01T13:06:38.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.282+0000 7f92f2898800 20 remove_watcher() i=3 2024-05-01T13:06:38.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.282+0000 7f92f2898800 20 remove_watcher() i=4 2024-05-01T13:06:38.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.283+0000 7f92f2898800 20 remove_watcher() i=5 2024-05-01T13:06:38.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.284+0000 7f92f2898800 20 remove_watcher() i=6 2024-05-01T13:06:38.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.285+0000 7f92f2898800 20 remove_watcher() i=7 2024-05-01T13:06:38.340 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.340 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.367+0000 7f8ae3c14800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.367+0000 7f8ae3c14800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.372+0000 7f8925ffb640 20 reqs_thread_entry: start 2024-05-01T13:06:38.428 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.428 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.456+0000 7f81b6a10800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.456+0000 7f81b6a10800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.460+0000 7f8074fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.463+0000 7f81b6a10800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:38.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.477+0000 7f81b6a10800 20 updating latest_epoch from 6 -> 7 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.567+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.569+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.569+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.570+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.570+0000 7fd457baf800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.570+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.570+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.570+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.571+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:38.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.571+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.571+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.571+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.572+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.572+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.572+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.572+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 zone MyZone found 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:7 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.573+0000 7fd457baf800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.574+0000 7fd457baf800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.574+0000 7fd457baf800 20 started sync module instance, tier type = 2024-05-01T13:06:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.574+0000 7fd457baf800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=0 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=1 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=2 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=3 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=4 2024-05-01T13:06:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=5 2024-05-01T13:06:38.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.584+0000 7fd457baf800 20 add_watcher() i=6 2024-05-01T13:06:38.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.585+0000 7fd457baf800 20 add_watcher() i=7 2024-05-01T13:06:38.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.585+0000 7fd457baf800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:38.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd30b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd457baf800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd457baf800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd457baf800 5 note: GC not initialized 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd30a7e4640 20 reqs_thread_entry: start 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd457baf800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd457baf800 20 Filter name: none 2024-05-01T13:06:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.587+0000 7fd27bfff640 20 reqs_thread_entry: start 2024-05-01T13:06:38.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.589+0000 7fd457baf800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:38.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.591+0000 7fd457baf800 20 updating latest_epoch from 7 -> 8 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.596+0000 7fd457baf800 4 Committed new epoch 8 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:38.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.598+0000 7fd30b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.600+0000 7fd457baf800 20 remove_watcher() i=0 2024-05-01T13:06:38.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.600+0000 7fd457baf800 2 removed watcher, disabling cache 2024-05-01T13:06:38.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.601+0000 7fd457baf800 20 remove_watcher() i=1 2024-05-01T13:06:38.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.602+0000 7fd457baf800 20 remove_watcher() i=2 2024-05-01T13:06:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.603+0000 7fd457baf800 20 remove_watcher() i=3 2024-05-01T13:06:38.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.604+0000 7fd457baf800 20 remove_watcher() i=4 2024-05-01T13:06:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.605+0000 7fd457baf800 20 remove_watcher() i=5 2024-05-01T13:06:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.606+0000 7fd457baf800 20 remove_watcher() i=6 2024-05-01T13:06:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.607+0000 7fd457baf800 20 remove_watcher() i=7 2024-05-01T13:06:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.716+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.718+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.718+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.719+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.719+0000 7fc68312e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.719+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.719+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.719+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.720+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:38.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.720+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.720+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.720+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.721+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.721+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.721+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.721+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 zone MyZone found 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:8 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.722+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.724 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:38.723+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:38.724 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:38.723+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:38.724 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:38.723+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:38.724 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:38.723+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.723+0000 7fc68312e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.723+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.724+0000 7fc68312e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.724+0000 7fc68312e800 20 started sync module instance, tier type = 2024-05-01T13:06:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.724+0000 7fc68312e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=0 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=1 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=2 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=3 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=4 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=5 2024-05-01T13:06:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.741+0000 7fc68312e800 20 add_watcher() i=6 2024-05-01T13:06:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.743+0000 7fc68312e800 20 add_watcher() i=7 2024-05-01T13:06:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.743+0000 7fc68312e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.745+0000 7fc5357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc68312e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc68312e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc68312e800 5 note: GC not initialized 2024-05-01T13:06:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc4affff640 20 reqs_thread_entry: start 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc68312e800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.746+0000 7fc68312e800 20 Filter name: none 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc4adffb640 20 reqs_thread_entry: start 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 : miss 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 info.flags=0x0 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 adding MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 to cache LRU end 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.6182933 : miss 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568798.6182933 info.flags=0x0 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 adding MyZone.rgw.meta+users.keys+1714568798.6182933 to cache LRU end 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.6182933 : hit (negative entry) 2024-05-01T13:06:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.747+0000 7fc68312e800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.6182933 : hit (negative entry) 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc68312e800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 info.flags=0x17 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc68312e800 10 moving MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 to cache LRU end 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc68312e800 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom_1$superman1714568798.6183019, ofs0, ns] 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc5627fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757306 cookie 93909211958688 notifier 7589 bl.length()=606 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc5627fc640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 info.flags=0x17 2024-05-01T13:06:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.749+0000 7fc5627fc640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom_1$superman1714568798.6183019 to cache LRU end 2024-05-01T13:06:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc68312e800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568798.6182933 info.flags=0x7 2024-05-01T13:06:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc68312e800 10 moving MyZone.rgw.meta+users.keys+1714568798.6182933 to cache LRU end 2024-05-01T13:06:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc68312e800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568798.6182933, ofs0, ns] 2024-05-01T13:06:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc561ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757412 cookie 93909211933744 notifier 7589 bl.length()=206 2024-05-01T13:06:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc561ffb640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568798.6182933 info.flags=0x7 2024-05-01T13:06:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.751+0000 7fc561ffb640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568798.6182933 to cache LRU end 2024-05-01T13:06:38.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.753+0000 7fc5357e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.755+0000 7fc68312e800 20 remove_watcher() i=0 2024-05-01T13:06:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.755+0000 7fc68312e800 2 removed watcher, disabling cache 2024-05-01T13:06:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.757+0000 7fc68312e800 20 remove_watcher() i=1 2024-05-01T13:06:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.758+0000 7fc68312e800 20 remove_watcher() i=2 2024-05-01T13:06:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.759+0000 7fc68312e800 20 remove_watcher() i=3 2024-05-01T13:06:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.760+0000 7fc68312e800 20 remove_watcher() i=4 2024-05-01T13:06:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.762+0000 7fc68312e800 20 remove_watcher() i=5 2024-05-01T13:06:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.763+0000 7fc68312e800 20 remove_watcher() i=6 2024-05-01T13:06:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.764+0000 7fc68312e800 20 remove_watcher() i=7 2024-05-01T13:06:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.877+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.879+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.879+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.880+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.880+0000 7f4f4312f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.880+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.880+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.880+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.881+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.881+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.882+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.882+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.882+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.882+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.883+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.883+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.883+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.883+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 20 zone MyZone found 2024-05-01T13:06:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:8 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.884+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.885+0000 7f4f4312f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.885+0000 7f4f4312f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.885+0000 7f4f4312f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.885+0000 7f4f4312f800 20 started sync module instance, tier type = 2024-05-01T13:06:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.885+0000 7f4f4312f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=0 2024-05-01T13:06:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=1 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=2 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=3 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=4 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=5 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.896+0000 7f4f4312f800 20 add_watcher() i=6 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.897+0000 7f4f4312f800 20 add_watcher() i=7 2024-05-01T13:06:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.897+0000 7f4f4312f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.898+0000 7f4df57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4f4312f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4f4312f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4f4312f800 5 note: GC not initialized 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4d6ffff640 20 reqs_thread_entry: start 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4f4312f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4f4312f800 20 Filter name: none 2024-05-01T13:06:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.899+0000 7f4d6dffb640 20 reqs_thread_entry: start 2024-05-01T13:06:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.911+0000 7f4df57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.913+0000 7f4f4312f800 20 remove_watcher() i=0 2024-05-01T13:06:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.913+0000 7f4f4312f800 2 removed watcher, disabling cache 2024-05-01T13:06:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.914+0000 7f4f4312f800 20 remove_watcher() i=1 2024-05-01T13:06:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.915+0000 7f4f4312f800 20 remove_watcher() i=2 2024-05-01T13:06:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.916+0000 7f4f4312f800 20 remove_watcher() i=3 2024-05-01T13:06:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.917+0000 7f4f4312f800 20 remove_watcher() i=4 2024-05-01T13:06:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.918+0000 7f4f4312f800 20 remove_watcher() i=5 2024-05-01T13:06:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.919+0000 7f4f4312f800 20 remove_watcher() i=6 2024-05-01T13:06:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:38.920+0000 7f4f4312f800 20 remove_watcher() i=7 2024-05-01T13:06:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.034+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.035+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.035+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.036+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.036+0000 7f139d746800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.036+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.036+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.037+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.037+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.037+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.038+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.038+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.038+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.038+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.039+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.039+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.039+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.039+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 zone MyZone found 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:8 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.040+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.041+0000 7f139d746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.041+0000 7f139d746800 20 started sync module instance, tier type = 2024-05-01T13:06:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.041+0000 7f139d746800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=0 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=1 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=2 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=3 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=4 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=5 2024-05-01T13:06:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.050+0000 7f139d746800 20 add_watcher() i=6 2024-05-01T13:06:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.052+0000 7f139d746800 20 add_watcher() i=7 2024-05-01T13:06:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.052+0000 7f139d746800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.053+0000 7f1251fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.053+0000 7f139d746800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.053+0000 7f139d746800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 5 note: GC not initialized 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f1250fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 20 Filter name: none 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f11ca7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 10 cache get: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 : miss 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 info.flags=0x0 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 10 adding MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 to cache LRU end 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.9334714 : miss 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.054+0000 7f139d746800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.055+0000 7f139d746800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.055+0000 7f139d746800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568798.9334714 info.flags=0x0 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.055+0000 7f139d746800 10 adding MyZone.rgw.meta+users.keys+1714568798.9334714 to cache LRU end 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.055+0000 7f139d746800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.9334714 : hit (negative entry) 2024-05-01T13:06:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.055+0000 7f139d746800 10 cache get: name=MyZone.rgw.meta+users.keys+1714568798.9334714 : hit (negative entry) 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f139d746800 10 cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 info.flags=0x17 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f139d746800 10 moving MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 to cache LRU end 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f139d746800 10 distributing notification oid=MyZone.rgw.control:notify.3 cni=[op: 0, obj: MyZone.rgw.meta:users.uid:kaboom_2$superman1714568798.9334822, ofs0, ns] 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f127effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757307 cookie 94475579450128 notifier 7645 bl.length()=602 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f127effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 info.flags=0x17 2024-05-01T13:06:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.056+0000 7f127effd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.uid+kaboom_2$superman1714568798.9334822 to cache LRU end 2024-05-01T13:06:39.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f139d746800 10 cache put: name=MyZone.rgw.meta+users.keys+1714568798.9334714 info.flags=0x7 2024-05-01T13:06:39.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f139d746800 10 moving MyZone.rgw.meta+users.keys+1714568798.9334714 to cache LRU end 2024-05-01T13:06:39.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f139d746800 10 distributing notification oid=MyZone.rgw.control:notify.0 cni=[op: 0, obj: MyZone.rgw.meta:users.keys:1714568798.9334714, ofs0, ns] 2024-05-01T13:06:39.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f127effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757413 cookie 94475579423056 notifier 7645 bl.length()=206 2024-05-01T13:06:39.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f127effd640 10 rgw watcher librados: cache put: name=MyZone.rgw.meta+users.keys+1714568798.9334714 info.flags=0x7 2024-05-01T13:06:39.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.058+0000 7f127effd640 10 rgw watcher librados: moving MyZone.rgw.meta+users.keys+1714568798.9334714 to cache LRU end 2024-05-01T13:06:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.059+0000 7f1251fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.061+0000 7f139d746800 20 remove_watcher() i=0 2024-05-01T13:06:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.061+0000 7f139d746800 2 removed watcher, disabling cache 2024-05-01T13:06:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.062+0000 7f139d746800 20 remove_watcher() i=1 2024-05-01T13:06:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.063+0000 7f139d746800 20 remove_watcher() i=2 2024-05-01T13:06:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.064+0000 7f139d746800 20 remove_watcher() i=3 2024-05-01T13:06:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.065+0000 7f139d746800 20 remove_watcher() i=4 2024-05-01T13:06:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.066+0000 7f139d746800 20 remove_watcher() i=5 2024-05-01T13:06:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.067+0000 7f139d746800 20 remove_watcher() i=6 2024-05-01T13:06:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.068+0000 7f139d746800 20 remove_watcher() i=7 2024-05-01T13:06:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.159+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.161+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.161+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.162+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.162+0000 7fdc8083f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.162+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.162+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.162+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.163+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.163+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.163+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.163+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.164+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.164+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.164+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.164+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 zone MyZone found 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:8 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.165+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.166+0000 7fdc8083f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.166+0000 7fdc8083f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.166+0000 7fdc8083f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.166+0000 7fdc8083f800 20 started sync module instance, tier type = 2024-05-01T13:06:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.166+0000 7fdc8083f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=0 2024-05-01T13:06:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=1 2024-05-01T13:06:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=2 2024-05-01T13:06:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=3 2024-05-01T13:06:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=4 2024-05-01T13:06:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=5 2024-05-01T13:06:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.175+0000 7fdc8083f800 20 add_watcher() i=6 2024-05-01T13:06:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.176+0000 7fdc8083f800 20 add_watcher() i=7 2024-05-01T13:06:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.176+0000 7fdc8083f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.178+0000 7fdb34fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.178+0000 7fdc8083f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.178+0000 7fdc8083f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.178+0000 7fdc8083f800 5 note: GC not initialized 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.178+0000 7fdaaf7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.179+0000 7fdc8083f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.179+0000 7fdc8083f800 20 Filter name: none 2024-05-01T13:06:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.179+0000 7fdaad7fa640 20 reqs_thread_entry: start 2024-05-01T13:06:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.190+0000 7fdb34fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.192+0000 7fdc8083f800 20 remove_watcher() i=0 2024-05-01T13:06:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.192+0000 7fdc8083f800 2 removed watcher, disabling cache 2024-05-01T13:06:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.193+0000 7fdc8083f800 20 remove_watcher() i=1 2024-05-01T13:06:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.194+0000 7fdc8083f800 20 remove_watcher() i=2 2024-05-01T13:06:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.195+0000 7fdc8083f800 20 remove_watcher() i=3 2024-05-01T13:06:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.196+0000 7fdc8083f800 20 remove_watcher() i=4 2024-05-01T13:06:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.197+0000 7fdc8083f800 20 remove_watcher() i=5 2024-05-01T13:06:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.198+0000 7fdc8083f800 20 remove_watcher() i=6 2024-05-01T13:06:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.199+0000 7fdc8083f800 20 remove_watcher() i=7 2024-05-01T13:06:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.395+0000 7f64bb131800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.395+0000 7f64bb131800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.399+0000 7f6374ff9640 20 reqs_thread_entry: start 2024-05-01T13:06:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.483+0000 7f6488319800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.483+0000 7f6488319800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.487+0000 7f62bbfff640 20 reqs_thread_entry: start 2024-05-01T13:06:39.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.490+0000 7f6488319800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:39.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.503+0000 7f6488319800 20 updating latest_epoch from 7 -> 8 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:39.558 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:39.558 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.594+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.595+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.595+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.596+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.596+0000 7f1dbb21f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.596+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.596+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.596+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.597+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.597+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.597+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.597+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.598+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.598+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.598+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.598+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 zone MyZone found 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:8 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.599+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.600+0000 7f1dbb21f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.600+0000 7f1dbb21f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.600+0000 7f1dbb21f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.600+0000 7f1dbb21f800 20 started sync module instance, tier type = 2024-05-01T13:06:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.600+0000 7f1dbb21f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=0 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=1 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=2 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=3 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=4 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=5 2024-05-01T13:06:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.609+0000 7f1dbb21f800 20 add_watcher() i=6 2024-05-01T13:06:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.610+0000 7f1dbb21f800 20 add_watcher() i=7 2024-05-01T13:06:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.610+0000 7f1dbb21f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.612+0000 7f1c6efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1dbb21f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1dbb21f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1dbb21f800 5 note: GC not initialized 2024-05-01T13:06:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1c6dfe3640 20 reqs_thread_entry: start 2024-05-01T13:06:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1dbb21f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1dbb21f800 20 Filter name: none 2024-05-01T13:06:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.613+0000 7f1bdf7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.615+0000 7f1dbb21f800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.617+0000 7f1dbb21f800 20 updating latest_epoch from 8 -> 9 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.623+0000 7f1dbb21f800 4 Committed new epoch 9 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.624+0000 7f1c6efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.626+0000 7f1dbb21f800 20 remove_watcher() i=0 2024-05-01T13:06:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.626+0000 7f1dbb21f800 2 removed watcher, disabling cache 2024-05-01T13:06:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.627+0000 7f1dbb21f800 20 remove_watcher() i=1 2024-05-01T13:06:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.629+0000 7f1dbb21f800 20 remove_watcher() i=2 2024-05-01T13:06:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.629+0000 7f1dbb21f800 20 remove_watcher() i=3 2024-05-01T13:06:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.630+0000 7f1dbb21f800 20 remove_watcher() i=4 2024-05-01T13:06:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.631+0000 7f1dbb21f800 20 remove_watcher() i=5 2024-05-01T13:06:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.633+0000 7f1dbb21f800 20 remove_watcher() i=6 2024-05-01T13:06:39.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.634+0000 7f1dbb21f800 20 remove_watcher() i=7 2024-05-01T13:06:39.803 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:39.803+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:39.803 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:39.803+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:39.803 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:39.803+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:39.803 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:39.803+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:39.960 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:39.960 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:39.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~51s 2024-05-01T13:06:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.993+0000 7fa7cb319800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.993+0000 7fa7cb319800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:39.998+0000 7fa60effd640 20 reqs_thread_entry: start 2024-05-01T13:06:40.062 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:40.062 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.094+0000 7f2702c0f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.094+0000 7f2702c0f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.099+0000 7f25c0fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.102+0000 7f2702c0f800 20 fork_period realm id=ecdbd3a5-899c-4649-8e14-01e69eca1c7f period id=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.116+0000 7f2702c0f800 20 updating latest_epoch from 8 -> 9 on period=ecdbd3a5-899c-4649-8e14-01e69eca1c7f:staging 2024-05-01T13:06:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:40.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.209+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.210+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.210+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.210+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:40.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.211+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.212+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=1107 2024-05-01T13:06:40.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.212+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.213+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.213+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.213+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.213+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 zone MyZone found 2024-05-01T13:06:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:9 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.214+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.215+0000 7f7940062800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.215+0000 7f7940062800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.215+0000 7f7940062800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.215+0000 7f7940062800 20 started sync module instance, tier type = 2024-05-01T13:06:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.215+0000 7f7940062800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:40.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=0 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=1 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=2 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=3 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=4 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=5 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.225+0000 7f7940062800 20 add_watcher() i=6 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.226+0000 7f7940062800 20 add_watcher() i=7 2024-05-01T13:06:40.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.226+0000 7f7940062800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:40.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f77ef7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f7940062800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f7940062800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f7940062800 5 note: GC not initialized 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f77ee7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.228+0000 7f7940062800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.229+0000 7f7940062800 20 Filter name: none 2024-05-01T13:06:40.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.229+0000 7f775ffff640 20 reqs_thread_entry: start 2024-05-01T13:06:40.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.230+0000 7f7940062800 20 commit_period realm ecdbd3a5-899c-4649-8e14-01e69eca1c7f period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.232+0000 7f7940062800 20 updating latest_epoch from 9 -> 10 on period=511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:40.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.241+0000 7f7940062800 4 Committed new epoch 10 for period 511d2184-89cf-4665-9417-968dd02e356b 2024-05-01T13:06:40.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.242+0000 7f77ef7fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:40.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.244+0000 7f7940062800 20 remove_watcher() i=0 2024-05-01T13:06:40.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.244+0000 7f7940062800 2 removed watcher, disabling cache 2024-05-01T13:06:40.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.245+0000 7f7940062800 20 remove_watcher() i=1 2024-05-01T13:06:40.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.246+0000 7f7940062800 20 remove_watcher() i=2 2024-05-01T13:06:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.247+0000 7f7940062800 20 remove_watcher() i=3 2024-05-01T13:06:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.248+0000 7f7940062800 20 remove_watcher() i=4 2024-05-01T13:06:40.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.249+0000 7f7940062800 20 remove_watcher() i=5 2024-05-01T13:06:40.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.250+0000 7f7940062800 20 remove_watcher() i=6 2024-05-01T13:06:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.251+0000 7f7940062800 20 remove_watcher() i=7 2024-05-01T13:06:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:40.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.343+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.344+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.344+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.345+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.345+0000 7f44d8a0e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.345+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.345+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.345+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.346+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.346+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.346+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.346+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.347+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.347+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.347+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.347+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 zone MyZone found 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.348+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.349+0000 7f44d8a0e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.349+0000 7f44d8a0e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.349+0000 7f44d8a0e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.349+0000 7f44d8a0e800 20 started sync module instance, tier type = 2024-05-01T13:06:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.349+0000 7f44d8a0e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=0 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=1 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=2 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=3 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=4 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=5 2024-05-01T13:06:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.359+0000 7f44d8a0e800 20 add_watcher() i=6 2024-05-01T13:06:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.361+0000 7f44d8a0e800 20 add_watcher() i=7 2024-05-01T13:06:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.361+0000 7f44d8a0e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.363+0000 7f438cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.363+0000 7f44d8a0e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.363+0000 7f44d8a0e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.363+0000 7f44d8a0e800 5 note: GC not initialized 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f43077fe640 20 reqs_thread_entry: start 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 20 Filter name: none 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f43057fa640 20 reqs_thread_entry: start 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x6 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:ERROR: could not remove topic: (2202) Unknown error 2202 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-05-01T13:06:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.364+0000 7f44d8a0e800 1 WARNING: topic migration in process. please try again later 2024-05-01T13:06:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.365+0000 7f438cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.367+0000 7f44d8a0e800 20 remove_watcher() i=0 2024-05-01T13:06:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.367+0000 7f44d8a0e800 2 removed watcher, disabling cache 2024-05-01T13:06:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.368+0000 7f44d8a0e800 20 remove_watcher() i=1 2024-05-01T13:06:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.369+0000 7f44d8a0e800 20 remove_watcher() i=2 2024-05-01T13:06:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.370+0000 7f44d8a0e800 20 remove_watcher() i=3 2024-05-01T13:06:40.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.371+0000 7f44d8a0e800 20 remove_watcher() i=4 2024-05-01T13:06:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.372+0000 7f44d8a0e800 20 remove_watcher() i=5 2024-05-01T13:06:40.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.373+0000 7f44d8a0e800 20 remove_watcher() i=6 2024-05-01T13:06:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:40.374+0000 7f44d8a0e800 20 remove_watcher() i=7 2024-05-01T13:06:40.479 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:40.479+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):730 WARNING: Cluster is is misconfigured! Zonegroup MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! 2024-05-01T13:06:40.479 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:40.479+0000 7f130d198640 -1 sync log trim: bool {anonymous}::sanity_check_endpoints(const DoutPrefixProvider*, rgw::sal::RadosStore*):740 ERROR: Cluster is is misconfigured! Zone MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) in Zonegroup MyZoneGroup ( 03460ef8-bf75-4e87-8999-38b335ab7ab6) in Realm id ( ecdbd3a5-899c-4649-8e14-01e69eca1c7f) has no endpoints! Trimming is impossible. 2024-05-01T13:06:40.479 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:40.479+0000 7f130d198640 -1 sync log trim: RGWCoroutine* create_meta_log_trim_cr(const DoutPrefixProvider*, rgw::sal::RadosStore*, RGWHTTPManager*, int, utime_t):759 ERROR: Cluster is is misconfigured! Refusing to trim. 2024-05-01T13:06:40.480 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:06:40.479+0000 7f130d198640 -1 rgw rados thread: Bailing out of trim thread! 2024-05-01T13:06:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.479+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.480+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.480+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.481+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.481+0000 7f6902f1f800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.481+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.481+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.481+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.482+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.482+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.483+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.484+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.484+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 20 zone MyZone found 2024-05-01T13:06:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.485+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.486+0000 7f6902f1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.486+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.486+0000 7f6902f1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.486+0000 7f6902f1f800 20 started sync module instance, tier type = 2024-05-01T13:06:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.486+0000 7f6902f1f800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=0 2024-05-01T13:06:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=1 2024-05-01T13:06:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=2 2024-05-01T13:06:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=3 2024-05-01T13:06:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=4 2024-05-01T13:06:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=5 2024-05-01T13:06:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.496+0000 7f6902f1f800 20 add_watcher() i=6 2024-05-01T13:06:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.498+0000 7f6902f1f800 20 add_watcher() i=7 2024-05-01T13:06:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.498+0000 7f6902f1f800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.499+0000 7f67b57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 5 note: GC not initialized 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f672ffff640 20 reqs_thread_entry: start 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 20 Filter name: none 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f672dffb640 20 reqs_thread_entry: start 2024-05-01T13:06:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.500+0000 7f6902f1f800 10 cache get: name=MyZone.rgw.log++pubsub. : miss 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 cache put: name=MyZone.rgw.log++pubsub. info.flags=0x0 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 adding MyZone.rgw.log++pubsub. to cache LRU end 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 cache get: name=MyZone.rgw.log++pubsub. : hit (negative entry) 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 cache get: name=MyZone.rgw.meta+topics+topic.:ashubx-27_topic_1 : miss 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 cache put: name=MyZone.rgw.meta+topics+topic.:ashubx-27_topic_1 info.flags=0x0 2024-05-01T13:06:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 adding MyZone.rgw.meta+topics+topic.:ashubx-27_topic_1 to cache LRU end 2024-05-01T13:06:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 1 failed to read topic info for name: ashubx-27_topic_1 tenant: , ret=-2 2024-05-01T13:06:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.501+0000 7f6902f1f800 10 WARNING: topic name:ashubx-27_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.502+0000 7f67b57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.504+0000 7f6902f1f800 20 remove_watcher() i=0 2024-05-01T13:06:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.504+0000 7f6902f1f800 2 removed watcher, disabling cache 2024-05-01T13:06:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.505+0000 7f6902f1f800 20 remove_watcher() i=1 2024-05-01T13:06:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.506+0000 7f6902f1f800 20 remove_watcher() i=2 2024-05-01T13:06:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.507+0000 7f6902f1f800 20 remove_watcher() i=3 2024-05-01T13:06:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.508+0000 7f6902f1f800 20 remove_watcher() i=4 2024-05-01T13:06:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.509+0000 7f6902f1f800 20 remove_watcher() i=5 2024-05-01T13:06:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.510+0000 7f6902f1f800 20 remove_watcher() i=6 2024-05-01T13:06:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.511+0000 7f6902f1f800 20 remove_watcher() i=7 2024-05-01T13:06:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:41.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.606+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.607+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.607+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.608+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.608+0000 7f979d413800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.608+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.608+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.608+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:41.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.609+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.610+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.610+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.610+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.610+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 20 zone MyZone found 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.611+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.612+0000 7f979d413800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.612+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.612+0000 7f979d413800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.612+0000 7f979d413800 20 started sync module instance, tier type = 2024-05-01T13:06:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.612+0000 7f979d413800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=0 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=1 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=2 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=3 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=4 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=5 2024-05-01T13:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.621+0000 7f979d413800 20 add_watcher() i=6 2024-05-01T13:06:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.623+0000 7f979d413800 20 add_watcher() i=7 2024-05-01T13:06:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.623+0000 7f979d413800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.624+0000 7f964dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.624+0000 7f979d413800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.624+0000 7f979d413800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 5 note: GC not initialized 2024-05-01T13:06:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f964cfe1640 20 reqs_thread_entry: start 2024-05-01T13:06:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 20 Filter name: none 2024-05-01T13:06:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f95c67fc640 20 reqs_thread_entry: start 2024-05-01T13:06:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_1 : miss 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom_1 info.flags=0x0 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 10 adding MyZone.rgw.log++pubsub.kaboom_1 to cache LRU end 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_1 : hit (negative entry) 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-28_topic_1 : miss 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.625+0000 7f979d413800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f979d413800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f979d413800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-28_topic_1 info.flags=0x0 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f979d413800 10 adding MyZone.rgw.meta+topics+topic.kaboom_1:ashubx-28_topic_1 to cache LRU end 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f979d413800 1 failed to read topic info for name: ashubx-28_topic_1 tenant: kaboom_1, ret=-2 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f979d413800 10 WARNING: topic name:ashubx-28_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.626+0000 7f964dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.628+0000 7f979d413800 20 remove_watcher() i=0 2024-05-01T13:06:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.628+0000 7f979d413800 2 removed watcher, disabling cache 2024-05-01T13:06:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.629+0000 7f979d413800 20 remove_watcher() i=1 2024-05-01T13:06:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.630+0000 7f979d413800 20 remove_watcher() i=2 2024-05-01T13:06:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.631+0000 7f979d413800 20 remove_watcher() i=3 2024-05-01T13:06:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.632+0000 7f979d413800 20 remove_watcher() i=4 2024-05-01T13:06:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.633+0000 7f979d413800 20 remove_watcher() i=5 2024-05-01T13:06:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.634+0000 7f979d413800 20 remove_watcher() i=6 2024-05-01T13:06:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.635+0000 7f979d413800 20 remove_watcher() i=7 2024-05-01T13:06:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.729+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.730+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.730+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.731+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.731+0000 7f467852d800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.731+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.732+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.732+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.732+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.732+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.733+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.734+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.734+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 zone MyZone found 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.735+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.736+0000 7f467852d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.736+0000 7f467852d800 20 started sync module instance, tier type = 2024-05-01T13:06:41.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.736+0000 7f467852d800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=0 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=1 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=2 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=3 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=4 2024-05-01T13:06:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=5 2024-05-01T13:06:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.745+0000 7f467852d800 20 add_watcher() i=6 2024-05-01T13:06:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.746+0000 7f467852d800 20 add_watcher() i=7 2024-05-01T13:06:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.746+0000 7f467852d800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:41.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.748+0000 7f452afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.748+0000 7f467852d800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.748+0000 7f467852d800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.748+0000 7f467852d800 5 note: GC not initialized 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f4529fdb640 20 reqs_thread_entry: start 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 20 Filter name: none 2024-05-01T13:06:41.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f449b7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:41.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_2 : miss 2024-05-01T13:06:41.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 cache put: name=MyZone.rgw.log++pubsub.kaboom_2 info.flags=0x0 2024-05-01T13:06:41.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 adding MyZone.rgw.log++pubsub.kaboom_2 to cache LRU end 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 cache get: name=MyZone.rgw.log++pubsub.kaboom_2 : hit (negative entry) 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 cache get: name=MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-29_topic_1 : miss 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 cache put: name=MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-29_topic_1 info.flags=0x0 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 adding MyZone.rgw.meta+topics+topic.kaboom_2:ashubx-29_topic_1 to cache LRU end 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 1 failed to read topic info for name: ashubx-29_topic_1 tenant: kaboom_2, ret=-2 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.749+0000 7f467852d800 10 WARNING: topic name:ashubx-29_topic_1 does not exist, deletion is a no-op: ret=-2 2024-05-01T13:06:41.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.750+0000 7f452afdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.752+0000 7f467852d800 20 remove_watcher() i=0 2024-05-01T13:06:41.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.752+0000 7f467852d800 2 removed watcher, disabling cache 2024-05-01T13:06:41.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.753+0000 7f467852d800 20 remove_watcher() i=1 2024-05-01T13:06:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.754+0000 7f467852d800 20 remove_watcher() i=2 2024-05-01T13:06:41.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.755+0000 7f467852d800 20 remove_watcher() i=3 2024-05-01T13:06:41.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.756+0000 7f467852d800 20 remove_watcher() i=4 2024-05-01T13:06:41.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.757+0000 7f467852d800 20 remove_watcher() i=5 2024-05-01T13:06:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.758+0000 7f467852d800 20 remove_watcher() i=6 2024-05-01T13:06:41.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.759+0000 7f467852d800 20 remove_watcher() i=7 2024-05-01T13:06:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.847+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.849+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.849+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.850+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.850+0000 7fee64b2e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.850+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.850+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.850+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.851+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.851+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.851+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.851+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.852+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.852+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.852+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.852+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 zone MyZone found 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.853+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.854+0000 7fee64b2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.854+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.854+0000 7fee64b2e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.854+0000 7fee64b2e800 20 started sync module instance, tier type = 2024-05-01T13:06:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.854+0000 7fee64b2e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=0 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=1 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=2 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=3 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=4 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=5 2024-05-01T13:06:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=6 2024-05-01T13:06:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 20 add_watcher() i=7 2024-05-01T13:06:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.864+0000 7fee64b2e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.866+0000 7fed16fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fee64b2e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fee64b2e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fee64b2e800 5 note: GC not initialized 2024-05-01T13:06:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fed15fe3640 20 reqs_thread_entry: start 2024-05-01T13:06:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fee64b2e800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fee64b2e800 20 Filter name: none 2024-05-01T13:06:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.867+0000 7fec877fe640 20 reqs_thread_entry: start 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got buckets.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got buckets.:ashubx-27_topic_created_v2 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v1 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v1 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got buckets.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v2 2024-05-01T13:06:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v1 2024-05-01T13:06:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.877+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.878+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=266 2024-05-01T13:06:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.878+0000 7fee64b2e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.878+0000 7fee64b2e800 20 rados_obj.operate() r=0 bl.length=266 2024-05-01T13:06:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.880+0000 7fed16fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.883+0000 7fee64b2e800 20 remove_watcher() i=0 2024-05-01T13:06:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.883+0000 7fee64b2e800 2 removed watcher, disabling cache 2024-05-01T13:06:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.884+0000 7fee64b2e800 20 remove_watcher() i=1 2024-05-01T13:06:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.884+0000 7fee64b2e800 20 remove_watcher() i=2 2024-05-01T13:06:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.886+0000 7fee64b2e800 20 remove_watcher() i=3 2024-05-01T13:06:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.886+0000 7fee64b2e800 20 remove_watcher() i=4 2024-05-01T13:06:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.887+0000 7fee64b2e800 20 remove_watcher() i=5 2024-05-01T13:06:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.888+0000 7fee64b2e800 20 remove_watcher() i=6 2024-05-01T13:06:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.889+0000 7fee64b2e800 20 remove_watcher() i=7 2024-05-01T13:06:41.944 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:41.944 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:41.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.980+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.981+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.981+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.982+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.982+0000 7fc309d0c800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.982+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.982+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.982+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.983+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.983+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.983+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.983+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.984+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.984+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.984+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.984+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 zone MyZone found 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.985+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.986+0000 7fc309d0c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.986+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.987+0000 7fc309d0c800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.987+0000 7fc309d0c800 20 started sync module instance, tier type = 2024-05-01T13:06:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.987+0000 7fc309d0c800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:41.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=0 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=1 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=2 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=3 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=4 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=5 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.996+0000 7fc309d0c800 20 add_watcher() i=6 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.997+0000 7fc309d0c800 20 add_watcher() i=7 2024-05-01T13:06:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.997+0000 7fc309d0c800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:41.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.999+0000 7fc1bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.999+0000 7fc309d0c800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.999+0000 7fc309d0c800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.999+0000 7fc309d0c800 5 note: GC not initialized 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:41.999+0000 7fc1bcfe1640 20 reqs_thread_entry: start 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.000+0000 7fc309d0c800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.000+0000 7fc309d0c800 20 Filter name: none 2024-05-01T13:06:42.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.000+0000 7fc1367fc640 20 reqs_thread_entry: start 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got buckets.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got buckets.:ashubx-27_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v1 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v1 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got buckets.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v2 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v1 2024-05-01T13:06:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.009+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.010+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=304 2024-05-01T13:06:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.010+0000 7fc309d0c800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.010+0000 7fc309d0c800 20 rados_obj.operate() r=0 bl.length=304 2024-05-01T13:06:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.013+0000 7fc1bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.015+0000 7fc309d0c800 20 remove_watcher() i=0 2024-05-01T13:06:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.015+0000 7fc309d0c800 2 removed watcher, disabling cache 2024-05-01T13:06:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.016+0000 7fc309d0c800 20 remove_watcher() i=1 2024-05-01T13:06:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.017+0000 7fc309d0c800 20 remove_watcher() i=2 2024-05-01T13:06:42.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.018+0000 7fc309d0c800 20 remove_watcher() i=3 2024-05-01T13:06:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.019+0000 7fc309d0c800 20 remove_watcher() i=4 2024-05-01T13:06:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.020+0000 7fc309d0c800 20 remove_watcher() i=5 2024-05-01T13:06:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.021+0000 7fc309d0c800 20 remove_watcher() i=6 2024-05-01T13:06:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.022+0000 7fc309d0c800 20 remove_watcher() i=7 2024-05-01T13:06:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.113+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.114+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.114+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.115+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.115+0000 7fdf2872e800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.115+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.115+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.115+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.116+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.117+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.117+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.117+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.117+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 20 zone MyZone found 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.118+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.119+0000 7fdf2872e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.119+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.119+0000 7fdf2872e800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.119+0000 7fdf2872e800 20 started sync module instance, tier type = 2024-05-01T13:06:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.119+0000 7fdf2872e800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=0 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=1 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=2 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=3 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=4 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=5 2024-05-01T13:06:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.128+0000 7fdf2872e800 20 add_watcher() i=6 2024-05-01T13:06:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.129+0000 7fdf2872e800 20 add_watcher() i=7 2024-05-01T13:06:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.129+0000 7fdf2872e800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.131+0000 7fdddafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdf2872e800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdf2872e800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdf2872e800 5 note: GC not initialized 2024-05-01T13:06:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fddd9fdb640 20 reqs_thread_entry: start 2024-05-01T13:06:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdf2872e800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdf2872e800 20 Filter name: none 2024-05-01T13:06:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.132+0000 7fdd4b7fe640 20 reqs_thread_entry: start 2024-05-01T13:06:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got buckets.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got buckets.:ashubx-27_topic_created_v2 2024-05-01T13:06:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v1 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.kaboom_2:ashubx-29_topic_created_v1 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.kaboom_1:ashubx-28_topic_created_v2 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got buckets.kaboom_2:ashubx-29_topic_created_v2 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v2 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 RGWRados::pool_iterate: got topic.:ashubx-27_topic_created_v1 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.143+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.144+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=304 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.144+0000 7fdf2872e800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.145+0000 7fdf2872e800 20 rados_obj.operate() r=0 bl.length=304 2024-05-01T13:06:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.146+0000 7fdddafdd640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.149+0000 7fdf2872e800 20 remove_watcher() i=0 2024-05-01T13:06:42.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.149+0000 7fdf2872e800 2 removed watcher, disabling cache 2024-05-01T13:06:42.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.150+0000 7fdf2872e800 20 remove_watcher() i=1 2024-05-01T13:06:42.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.151+0000 7fdf2872e800 20 remove_watcher() i=2 2024-05-01T13:06:42.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.152+0000 7fdf2872e800 20 remove_watcher() i=3 2024-05-01T13:06:42.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.153+0000 7fdf2872e800 20 remove_watcher() i=4 2024-05-01T13:06:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.153+0000 7fdf2872e800 20 remove_watcher() i=5 2024-05-01T13:06:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.155+0000 7fdf2872e800 20 remove_watcher() i=6 2024-05-01T13:06:42.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.156+0000 7fdf2872e800 20 remove_watcher() i=7 2024-05-01T13:06:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:42.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.247+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.248+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.248+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.249+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.249+0000 7f4ed8c10800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.249+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.249+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.249+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.250+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.250+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.250+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.250+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.251+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.251+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.251+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.251+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 zone MyZone found 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.252+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.253+0000 7f4ed8c10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.253+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.253+0000 7f4ed8c10800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.253+0000 7f4ed8c10800 20 started sync module instance, tier type = 2024-05-01T13:06:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.253+0000 7f4ed8c10800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=0 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=1 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=2 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=3 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=4 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=5 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.263+0000 7f4ed8c10800 20 add_watcher() i=6 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.264+0000 7f4ed8c10800 20 add_watcher() i=7 2024-05-01T13:06:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.264+0000 7f4ed8c10800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4d8a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4ed8c10800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4ed8c10800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4ed8c10800 5 note: GC not initialized 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4d897e2640 20 reqs_thread_entry: start 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.266+0000 7f4ed8c10800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.267+0000 7f4ed8c10800 20 Filter name: none 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.267+0000 7f4cfaffd640 20 reqs_thread_entry: start 2024-05-01T13:06:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.267+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.267+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=187 2024-05-01T13:06:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.267+0000 7f4ed8c10800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.268+0000 7f4ed8c10800 20 rados_obj.operate() r=0 bl.length=401 2024-05-01T13:06:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.268+0000 7f4ed8c10800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.269+0000 7f4d8a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.270+0000 7f4ed8c10800 20 remove_watcher() i=0 2024-05-01T13:06:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.270+0000 7f4ed8c10800 2 removed watcher, disabling cache 2024-05-01T13:06:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.271+0000 7f4ed8c10800 20 remove_watcher() i=1 2024-05-01T13:06:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.272+0000 7f4ed8c10800 20 remove_watcher() i=2 2024-05-01T13:06:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.273+0000 7f4ed8c10800 20 remove_watcher() i=3 2024-05-01T13:06:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.274+0000 7f4ed8c10800 20 remove_watcher() i=4 2024-05-01T13:06:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.276+0000 7f4ed8c10800 20 remove_watcher() i=5 2024-05-01T13:06:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.277+0000 7f4ed8c10800 20 remove_watcher() i=6 2024-05-01T13:06:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.278+0000 7f4ed8c10800 20 remove_watcher() i=7 2024-05-01T13:06:42.335 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:42.335 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.371+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.373+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.373+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.373+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.373+0000 7f5aa7a62800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.374+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.374+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.374+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.375+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.376+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.376+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.376+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.376+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 20 zone MyZone found 2024-05-01T13:06:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.377+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.378+0000 7f5aa7a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.378+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.378+0000 7f5aa7a62800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.378+0000 7f5aa7a62800 20 started sync module instance, tier type = 2024-05-01T13:06:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.378+0000 7f5aa7a62800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=0 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=1 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=2 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=3 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=4 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=5 2024-05-01T13:06:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.387+0000 7f5aa7a62800 20 add_watcher() i=6 2024-05-01T13:06:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.388+0000 7f5aa7a62800 20 add_watcher() i=7 2024-05-01T13:06:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.388+0000 7f5aa7a62800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.390+0000 7f595b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.390+0000 7f5aa7a62800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.390+0000 7f5aa7a62800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.390+0000 7f5aa7a62800 5 note: GC not initialized 2024-05-01T13:06:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.390+0000 7f595a7e4640 20 reqs_thread_entry: start 2024-05-01T13:06:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f5aa7a62800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f5aa7a62800 20 Filter name: none 2024-05-01T13:06:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f58cbfff640 20 reqs_thread_entry: start 2024-05-01T13:06:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=231 2024-05-01T13:06:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.391+0000 7f5aa7a62800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.392+0000 7f5aa7a62800 20 rados_obj.operate() r=0 bl.length=453 2024-05-01T13:06:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.392+0000 7f5aa7a62800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.393+0000 7f595b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.395+0000 7f5aa7a62800 20 remove_watcher() i=0 2024-05-01T13:06:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.395+0000 7f5aa7a62800 2 removed watcher, disabling cache 2024-05-01T13:06:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.396+0000 7f5aa7a62800 20 remove_watcher() i=1 2024-05-01T13:06:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.397+0000 7f5aa7a62800 20 remove_watcher() i=2 2024-05-01T13:06:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.398+0000 7f5aa7a62800 20 remove_watcher() i=3 2024-05-01T13:06:42.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.399+0000 7f5aa7a62800 20 remove_watcher() i=4 2024-05-01T13:06:42.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.400+0000 7f5aa7a62800 20 remove_watcher() i=5 2024-05-01T13:06:42.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.401+0000 7f5aa7a62800 20 remove_watcher() i=6 2024-05-01T13:06:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.402+0000 7f5aa7a62800 20 remove_watcher() i=7 2024-05-01T13:06:42.459 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2024-05-01T13:06:42.459 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2024-05-01T13:06:42.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.495+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 realm MyRealm ecdbd3a5-899c-4649-8e14-01e69eca1c7f 2024-05-01T13:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.497+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.498+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=10 2024-05-01T13:06:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.498+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=1126 2024-05-01T13:06:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=107 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.499+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=46 2024-05-01T13:06:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=1135 2024-05-01T13:06:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 zone MyZone found 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 4 Realm: MyRealm (ecdbd3a5-899c-4649-8e14-01e69eca1c7f) 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 4 ZoneGroup: MyZoneGroup (03460ef8-bf75-4e87-8999-38b335ab7ab6) 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 4 Zone: MyZone (b3b7ffad-8c5b-4f40-9e07-35170fac4823) 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 4 using period configuration: 511d2184-89cf-4665-9417-968dd02e356b:10 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 period zonegroup name MyZoneGroup 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 using current period zonegroup MyZoneGroup 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.500+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.501+0000 7f5f7fb29800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.501+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.502+0000 7f5f7fb29800 20 rados_obj.operate() r=-2 bl.length=0 2024-05-01T13:06:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.502+0000 7f5f7fb29800 20 started sync module instance, tier type = 2024-05-01T13:06:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.502+0000 7f5f7fb29800 20 started zone id=b3b7ffad-8c5b-4f40-9e07-35170fac4823 (name=MyZone) with tier type = 2024-05-01T13:06:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=0 2024-05-01T13:06:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=1 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=2 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=3 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=4 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=5 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.514+0000 7f5f7fb29800 20 add_watcher() i=6 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.515+0000 7f5f7fb29800 20 add_watcher() i=7 2024-05-01T13:06:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.515+0000 7f5f7fb29800 2 all 8 watchers are set, enabling cache 2024-05-01T13:06:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.516+0000 7f5e31fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 5 note: GC not initialized 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5e30fe1640 20 reqs_thread_entry: start 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 20 init_complete bucket index max shards: 11 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 20 Filter name: none 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5daa7fc640 20 reqs_thread_entry: start 2024-05-01T13:06:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.517+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.518+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=231 2024-05-01T13:06:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.518+0000 7f5f7fb29800 20 rados->read ofs=0 len=0 2024-05-01T13:06:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.518+0000 7f5f7fb29800 20 rados_obj.operate() r=0 bl.length=453 2024-05-01T13:06:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.518+0000 7f5f7fb29800 20 couldn't put binfo cache entry, might have raced with data changes 2024-05-01T13:06:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.519+0000 7f5e31fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-05-01T13:06:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.522+0000 7f5f7fb29800 20 remove_watcher() i=0 2024-05-01T13:06:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.522+0000 7f5f7fb29800 2 removed watcher, disabling cache 2024-05-01T13:06:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.523+0000 7f5f7fb29800 20 remove_watcher() i=1 2024-05-01T13:06:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.524+0000 7f5f7fb29800 20 remove_watcher() i=2 2024-05-01T13:06:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.525+0000 7f5f7fb29800 20 remove_watcher() i=3 2024-05-01T13:06:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.526+0000 7f5f7fb29800 20 remove_watcher() i=4 2024-05-01T13:06:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.527+0000 7f5f7fb29800 20 remove_watcher() i=5 2024-05-01T13:06:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.527+0000 7f5f7fb29800 20 remove_watcher() i=6 2024-05-01T13:06:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-05-01T13:06:42.528+0000 7f5f7fb29800 20 remove_watcher() i=7 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-15?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:pw+V4cykcAqXCL2U5YWfZe+HCJ4=\r\nDate: Wed, 01 May 2024 12:59:14 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000a377cb008d84a482-0066323ca2-4559-MyZone 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 12:59:14 GMT 2024-05-01T13:06:42.561 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 26.53517723083496 milliseconds 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-16?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:l/V0X7hY6VernrDOu/rhhfYmhCs=\r\nDate: Wed, 01 May 2024 13:01:15 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000e2d85848c7ba5946-0066323d1b-4559-MyZone 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:01:15 GMT 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 24.76114511489868 milliseconds 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + async http notification is: 2.0671963691711426 milliseconds 2024-05-01T13:06:42.562 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-18?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:T6Ygon3SB8Yerp+BV+X1RCp4Bzg=\r\nDate: Wed, 01 May 2024 13:04:48 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000001079f2dfaa88e5a7-0066323df0-4559-MyZone 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:04:48 GMT 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 23.865337371826172 milliseconds 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:average time for deletion + async http notification is: 1.7786431312561035 milliseconds 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-19?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568689.032559:TNRxIAS4GiCOJ1zRHpwTIhKnJnc=\r\nDate: Wed, 01 May 2024 13:05:43 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.563 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000651bfe7a8d51eedd-0066323e27-4559-MyZone 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:05:43 GMT 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-20?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568743.7135923:GzFzJQPodxmUF+tGL5kKsNdb2dg=\r\nDate: Wed, 01 May 2024 13:05:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000fbb0fce25294d547-0066323e28-4559-MyZone 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.564 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:05:44 GMT 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-21?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568744.1266594:nG//GU3PvVRW8AoAaeXpwXgAjGM=\r\nDate: Wed, 01 May 2024 13:05:44 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000b933ca6a898fcd71-0066323e28-4559-MyZone 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:05:44 GMT 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 24.750518798828125 milliseconds 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + async http notification is: 24.80292320251465 milliseconds 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-22?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:4c3LoBSANeyMjPXTdgavXmIGAyo=\r\nDate: Wed, 01 May 2024 13:06:28 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.565 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000001074879b631417fa-0066323e54-7188-MyZone 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:28 GMT 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 27.422475814819336 milliseconds 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:average time for creation + http notification is: 26.244544982910156 milliseconds 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-23?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:CN5yBuizaTGIQe/tqzkx7nqHI1g=\r\nDate: Wed, 01 May 2024 13:06:31 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000a85332283ab9b168-0066323e57-7043-MyZone 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.566 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:31 GMT 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:('', 0) 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:('', 0) 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:('', 0) 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:('', 0) 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:migration in process... error: 154 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-24?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:kQVbW7e4dCRBTJ7XlRK+vzMTxaI=\r\nDate: Wed, 01 May 2024 13:06:37 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000009770143fa0c4bfbb-0066323e5d-7815-MyZone 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:37 GMT 2024-05-01T13:06:42.567 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-25?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568792.5982735:kArC7PhaGs50voTVhsq2DLbOweE=\r\nDate: Wed, 01 May 2024 13:06:37 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000c524d1bc53fb2f0c-0066323e5d-7815-MyZone 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:37 GMT 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-26?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568792.8651698:2UkVvvtIAvd0ERO2foDbqEq5cME=\r\nDate: Wed, 01 May 2024 13:06:37 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000004dd73c5b241b59be-0066323e5d-7815-MyZone 2024-05-01T13:06:42.568 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:38 GMT 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:('', 154) 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-27?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS NRWJLBIBJRWYBMQBMFAB:XhT/bxjDcm3EQ+D9fowtxTkOy+U=\r\nDate: Wed, 01 May 2024 13:06:42 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx00000661abeb63353f395-0066323e62-8289-MyZone 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:42 GMT 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-28?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568798.6182933:a0JSn0z0jnTvWWvxigp17lWuSE0=\r\nDate: Wed, 01 May 2024 13:06:42 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.569 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx0000066200af26adef298-0066323e62-8289-MyZone 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: Server: Ceph Object Gateway (squid) 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: Content-Length: 0 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: Date: Wed, 01 May 2024 13:06:42 GMT 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: Connection: Keep-Alive 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:send: b'DELETE /ashubx-29?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS 1714568798.9334714:W61jT6R8vYrpvstigMcjBRHBrH8=\r\nDate: Wed, 01 May 2024 13:06:42 +0000\r\nHost: smithi050.front.sepia.ceph.com:80\r\n\r\n' 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:Zero length chunk ignored 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-05-01T13:06:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:header: x-amz-request-id: tx000001aca7aa55b4a8178-0066323e62-8289-MyZone 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:ok 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:====================================================================== 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:FAIL: test persistent topic stats 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:---------------------------------------------------------------------- 2024-05-01T13:06:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:Traceback (most recent call last): 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/case.py", line 198, in runTest 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: self.test(*self.arg) 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 3043, in test_ps_s3_persistent_topic_stats 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: wait_for_queue_to_drain(topic_name) 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2961, in wait_for_queue_to_drain 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr: assert_equal(entries, 0) 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:AssertionError: 200 != 0 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:-------------------- >> begin captured logging << -------------------- 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Using access key provided by client. 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Using secret key provided by client. 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/ 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/ 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/ 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {} 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/ 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:YZ7VE1Gib258wfkjCHeEi5MjdBw= 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:YZ7VE1Gib258wfkjCHeEi5MjdBw=', 'Content-Length': '0'} 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004ad67ad4e2dbd3ee-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Wed, 01 May 2024 12:54:55 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event creating-client-class.sns: calling handler 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Creating a regex based endpoint for sns, MyZoneGroup 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Setting sns timeout as (60, 60) 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.client: DEBUG: Registering retry handlers for service: sns 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.sns.CreateTopic: calling handler 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Region': 'MyZoneGroup', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://smithi050.front.sepia.ceph.com:80'} 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Endpoint provider result: http://smithi050.front.sepia.ceph.com:80 2024-05-01T13:06:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.95 md/Botocore#1.34.95 ua/2.0 os/linux#5.14.0-437.el9.x86_64 md/arch#x86_64 lang/python#3.9.18 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.95'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'ashubx-14_topic', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'http://localhost:19588', 'Attributes.entry.2.key': 'persistent', 'Attributes.entry.2.value': 'true'}, 'url': 'http://smithi050.front.sepia.ceph.com:80/', 'context': {'client_region': 'MyZoneGroup', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler > 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event choose-signer.sns.CreateTopic: calling handler 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:POST 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:/ 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:host:smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:x-amz-date:20240501T125455Z 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:content-type;host;x-amz-date 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:49ab3b9251d8fd2d5a2227d02678ea73984b205f28415d60af0ff2fe2217caf8 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: StringToSign: 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:AWS4-HMAC-SHA256 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:20240501T125455Z 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:20240501/MyZoneGroup/sns/aws4_request 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:8b2115c164c0dc90c76ab9357dbe754fe7af4b7ebfc3155f008feb98482db0f3 2024-05-01T13:06:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: Signature: 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:9b80ce7fb0a3cab5abb2315d0161c15ca96e4267c0807fb2327486d873f899db 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi050.front.sepia.ceph.com:80 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:urllib3.connectionpool: DEBUG: http://smithi050.front.sepia.ceph.com:80 "POST / HTTP/1.1" 200 314 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000004d8a3375224914f3-0066323b9f-4559-MyZone', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '314', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Connection': 'Keep-Alive'} 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.parsers: DEBUG: Response body: 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:b'arn:aws:sns:MyZoneGroup::ashubx-14_topicb3b7ffad-8c5b-4f40-9e07-35170fac4823.4559.5587334865885926643' 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event needs-retry.sns.CreateTopic: calling handler 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-05-01T13:06:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7fe3f0d85430> 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.utils: DEBUG: Registering S3 region redirector handler 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.utils: DEBUG: Registering S3Express Identity Resolver 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-endpoint-resolution.s3: calling handler > 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Calling endpoint provider with parameters: {'Bucket': 'ashubx-14', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://smithi050.front.sepia.ceph.com:80', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True, 'UseS3ExpressControlEndpoint': True} 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Endpoint provider result: http://smithi050.front.sepia.ceph.com:80/ashubx-14 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.regions: DEBUG: Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.34.95 md/Botocore#1.34.95 ua/2.0 os/linux#5.14.0-437.el9.x86_64 md/arch#x86_64 lang/python#3.9.18 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.95'}, 'body': b'ashubx-14_notifarn:aws:sns:MyZoneGroup::ashubx-14_topic', 'auth_path': '/ashubx-14/', 'url': 'http://smithi050.front.sepia.ceph.com:80/ashubx-14?notification', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'ashubx-14', 'params': {'Bucket': 'ashubx-14', 'NotificationConfiguration': {'TopicConfigurations': [{'Id': 'ashubx-14_notif', 'TopicArn': 'arn:aws:sns:MyZoneGroup::ashubx-14_topic', 'Events': []}]}}}, 'input_params': {'Bucket': 'ashubx-14'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}} 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:notification= 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:host:smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:x-amz-content-sha256:b6046c39d11a99b922ad5bb7d907ed3fc70fd52af726c5b9383f6ce3c9daec98 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:x-amz-date:20240501T125455Z 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:host;x-amz-content-sha256;x-amz-date 2024-05-01T13:06:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:b6046c39d11a99b922ad5bb7d907ed3fc70fd52af726c5b9383f6ce3c9daec98 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: StringToSign: 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:AWS4-HMAC-SHA256 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:20240501T125455Z 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:20240501/us-east-1/s3/aws4_request 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:9c7dbab6a67bddfd604a5125c1ca75519455e65e1ba813e2944f18f5fb149b67 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.auth: DEBUG: Signature: 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:f8bf0f09ef3bf82f2867635e76fda483dea13c900a69acf7f10d06804bdaf407 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi050.front.sepia.ceph.com:80 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:urllib3.connectionpool: DEBUG: http://smithi050.front.sepia.ceph.com:80 "PUT /ashubx-14?notification HTTP/1.1" 200 0 2024-05-01T13:06:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000031e480f35a0a975c-0066323b9f-4559-MyZone', 'Server': 'Ceph Object Gateway (squid)', 'Content-Length': '0', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Connection': 'Keep-Alive'} 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.parsers: DEBUG: Response body: 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:b'' 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:bucket_notification.tests: DEBUG: running command: /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/../test-rgw-call.sh call_rgw_admin noname topic stats --topic ashubx-14_topic 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-0 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-0 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-0 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ulEl9+TQmrZEDDCM+Mk1Ug==', 'Content-Length': '3008970', 'Expect': '100-Continue'} 2024-05-01T13:06:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:ulEl9+TQmrZEDDCM+Mk1Ug== 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-0 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:4LM3ODqBU6V4HP8rxbEXsoeyeDM= 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ulEl9+TQmrZEDDCM+Mk1Ug==', 'Content-Length': '3008970', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:4LM3ODqBU6V4HP8rxbEXsoeyeDM='} 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-1 2024-05-01T13:06:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-1 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-1 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'C80KBkFTAXeqeeHMV98dUg==', 'Content-Length': '3012044', 'Expect': '100-Continue'} 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:C80KBkFTAXeqeeHMV98dUg== 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-1 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:6e02QaJhyZEOJ8N8BW5DnHEHjeQ= 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'C80KBkFTAXeqeeHMV98dUg==', 'Content-Length': '3012044', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:6e02QaJhyZEOJ8N8BW5DnHEHjeQ='} 2024-05-01T13:06:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-2 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-2 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-2 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'am6YYcigDDKS0SKWQOSUpQ==', 'Content-Length': '3010843', 'Expect': '100-Continue'} 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:am6YYcigDDKS0SKWQOSUpQ== 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-2 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:Ysx3Nprdp/R8jh0219OzTchGowg= 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'am6YYcigDDKS0SKWQOSUpQ==', 'Content-Length': '3010843', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:Ysx3Nprdp/R8jh0219OzTchGowg='} 2024-05-01T13:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"ba5125f7e4d09ab6440c308cf8c93552"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000111abcc8a0557ea6-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:55 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-3 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-3 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-3 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '29LbATNdWAdCAbmbFbZ7Kg==', 'Content-Length': '3012249', 'Expect': '100-Continue'} 2024-05-01T13:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:29LbATNdWAdCAbmbFbZ7Kg== 2024-05-01T13:06:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-3 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:xtZkdQofUKDtSLQPBssnD+jyPr4= 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '29LbATNdWAdCAbmbFbZ7Kg==', 'Content-Length': '3012249', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:xtZkdQofUKDtSLQPBssnD+jyPr4='} 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-4 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-4 2024-05-01T13:06:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-4 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'QVYFhqSzJWE1yWBdnIv8fw==', 'Content-Length': '3009591', 'Expect': '100-Continue'} 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.721 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:QVYFhqSzJWE1yWBdnIv8fw== 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-4 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:ZnunX/gb/tyGCibvY+B0p0em/MQ= 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'QVYFhqSzJWE1yWBdnIv8fw==', 'Content-Length': '3009591', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:ZnunX/gb/tyGCibvY+B0p0em/MQ='} 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-5 2024-05-01T13:06:42.723 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-5 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-5 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CeBxUoBQW4zlGmCGkZdyLg==', 'Content-Length': '3008696', 'Expect': '100-Continue'} 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.724 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:CeBxUoBQW4zlGmCGkZdyLg== 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-5 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:2AFZbkEEvy+2qDCAbLqqgAsNU7M= 2024-05-01T13:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CeBxUoBQW4zlGmCGkZdyLg==', 'Content-Length': '3008696', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:2AFZbkEEvy+2qDCAbLqqgAsNU7M='} 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-6 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-6 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-6 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7rX/pIRAlznRlUgnsD1nRA==', 'Content-Length': '3008198', 'Expect': '100-Continue'} 2024-05-01T13:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:7rX/pIRAlznRlUgnsD1nRA== 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-6 2024-05-01T13:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:LVjtgfLNWVt+v8R5hnnJnYyGn+c= 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7rX/pIRAlznRlUgnsD1nRA==', 'Content-Length': '3008198', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:LVjtgfLNWVt+v8R5hnnJnYyGn+c='} 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-7 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-7 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-7 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vme8hoK/A3KsJoOwIfvcEg==', 'Content-Length': '3010776', 'Expect': '100-Continue'} 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:vme8hoK/A3KsJoOwIfvcEg== 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-7 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:dO7GVbS388OEj+3CqZITyHDartA= 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vme8hoK/A3KsJoOwIfvcEg==', 'Content-Length': '3010776', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:dO7GVbS388OEj+3CqZITyHDartA='} 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0bcd0a0641530177aa79e1cc57df1d52"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af52d86a12b38daa-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:55 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-8 2024-05-01T13:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-8 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-8 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YozkC20vzE0Rx4zDVuTmDQ==', 'Content-Length': '3011886', 'Expect': '100-Continue'} 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:YozkC20vzE0Rx4zDVuTmDQ== 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-8 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:r+xligp5rhtt+cr9X7uMrtmEdxo= 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YozkC20vzE0Rx4zDVuTmDQ==', 'Content-Length': '3011886', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:r+xligp5rhtt+cr9X7uMrtmEdxo='} 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-9 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-9 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-9 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UA4iYv2+I8OTZwnh3L4FtA==', 'Content-Length': '3012174', 'Expect': '100-Continue'} 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:UA4iYv2+I8OTZwnh3L4FtA== 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-9 2024-05-01T13:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:KOemfeep/U43AyqlgkWTz2q3foo= 2024-05-01T13:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UA4iYv2+I8OTZwnh3L4FtA==', 'Content-Length': '3012174', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:KOemfeep/U43AyqlgkWTz2q3foo='} 2024-05-01T13:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-10 2024-05-01T13:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-10 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-10 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'z5RyoRVpCvueZCdWQgpLDg==', 'Content-Length': '3010727', 'Expect': '100-Continue'} 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:z5RyoRVpCvueZCdWQgpLDg== 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-10 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:f1MbTElTPhPaSj1j/I6x48eBwwI= 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'z5RyoRVpCvueZCdWQgpLDg==', 'Content-Length': '3010727', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:f1MbTElTPhPaSj1j/I6x48eBwwI='} 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-11 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-11 2024-05-01T13:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-11 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'c2NkmXEFRmAIvkA5tW0DvQ==', 'Content-Length': '3007117', 'Expect': '100-Continue'} 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:c2NkmXEFRmAIvkA5tW0DvQ== 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-11 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:0NI/tonJX5/cv+7MemHqpFjVNRE= 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'c2NkmXEFRmAIvkA5tW0DvQ==', 'Content-Length': '3007117', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:0NI/tonJX5/cv+7MemHqpFjVNRE='} 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6a6e9861c8a00c3292d1229640e494a5"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be94a970a9b09e57-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:55 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-12 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-12 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-12 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'P/Oe1AKyGNDqa5IsZag27Q==', 'Content-Length': '3011591', 'Expect': '100-Continue'} 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:P/Oe1AKyGNDqa5IsZag27Q== 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:55 GMT 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-12 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:0X4ls5UKEYmROK5qehPv3eb3E74= 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'P/Oe1AKyGNDqa5IsZag27Q==', 'Content-Length': '3011591', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:55 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:0X4ls5UKEYmROK5qehPv3eb3E74='} 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-13 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-13 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-13 2024-05-01T13:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vg8eHmgBIvDB1sQt9+yZKQ==', 'Content-Length': '3012412', 'Expect': '100-Continue'} 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:vg8eHmgBIvDB1sQt9+yZKQ== 2024-05-01T13:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-13 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:/YLEC8plqcoBQwgf29bxVI+WUrg= 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vg8eHmgBIvDB1sQt9+yZKQ==', 'Content-Length': '3012412', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:/YLEC8plqcoBQwgf29bxVI+WUrg='} 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-16 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-16 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-16 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VGYgk/wizaSUFfeN/ow2XQ==', 'Content-Length': '3009164', 'Expect': '100-Continue'} 2024-05-01T13:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:VGYgk/wizaSUFfeN/ow2XQ== 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-16 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:S5GOk4uBUu9pPf7pBpiWQbyX3Hg= 2024-05-01T13:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VGYgk/wizaSUFfeN/ow2XQ==', 'Content-Length': '3009164', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:S5GOk4uBUu9pPf7pBpiWQbyX3Hg='} 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-14 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-14 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-14 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Fq+X0qJ3kpZzNYkMBFXRlg==', 'Content-Length': '3011761', 'Expect': '100-Continue'} 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:Fq+X0qJ3kpZzNYkMBFXRlg== 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-14 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:8FuPp0MTmZmY2x4YBj0E0Cz2JWU= 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Fq+X0qJ3kpZzNYkMBFXRlg==', 'Content-Length': '3011761', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:8FuPp0MTmZmY2x4YBj0E0Cz2JWU='} 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dbd2db01335d58074201b99b15b67b2a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec8bcfafebb4dda5-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-15 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-15 2024-05-01T13:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-15 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6mBaJi0nrD0rKd42MLHP0Q==', 'Content-Length': '3014262', 'Expect': '100-Continue'} 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:6mBaJi0nrD0rKd42MLHP0Q== 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-15 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:S7aDscjq/XaZMe2eze0bmKwOcCA= 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6mBaJi0nrD0rKd42MLHP0Q==', 'Content-Length': '3014262', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:S7aDscjq/XaZMe2eze0bmKwOcCA='} 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-17 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-17 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-17 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vYLNrslP5FXQNe3E6ulLCw==', 'Content-Length': '3011267', 'Expect': '100-Continue'} 2024-05-01T13:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:vYLNrslP5FXQNe3E6ulLCw== 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-17 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:tldq5GoFAEOCztVBRdl6sLCq03E= 2024-05-01T13:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vYLNrslP5FXQNe3E6ulLCw==', 'Content-Length': '3011267', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:tldq5GoFAEOCztVBRdl6sLCq03E='} 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-18 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-18 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-18 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK51r/T4B0RXH5I4E/mo/g==', 'Content-Length': '3008619', 'Expect': '100-Continue'} 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:AK51r/T4B0RXH5I4E/mo/g== 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-18 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:CO6jCjer4UiYwRJqzn2uTXi8duk= 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK51r/T4B0RXH5I4E/mo/g==', 'Content-Length': '3008619', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:CO6jCjer4UiYwRJqzn2uTXi8duk='} 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-19 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-19 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-19 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uhlQR33SZhbCUw7IdTs8MA==', 'Content-Length': '3011485', 'Expect': '100-Continue'} 2024-05-01T13:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:uhlQR33SZhbCUw7IdTs8MA== 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-19 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:gdVEr8gVlteI1v7pbklidvdZEyU= 2024-05-01T13:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uhlQR33SZhbCUw7IdTs8MA==', 'Content-Length': '3011485', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:gdVEr8gVlteI1v7pbklidvdZEyU='} 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-20 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-20 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-20 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ssALJcssJxwzW424YW1Yvg==', 'Content-Length': '3011117', 'Expect': '100-Continue'} 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:ssALJcssJxwzW424YW1Yvg== 2024-05-01T13:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-20 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:l0FB+lqmCDe0F6JabmRjpx24DMM= 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ssALJcssJxwzW424YW1Yvg==', 'Content-Length': '3011117', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:l0FB+lqmCDe0F6JabmRjpx24DMM='} 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"41560586a4b3256135c9605d9c8bfc7f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c875c2bab4b31784-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"09e0715280505b8ce51a60869197722e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5052e0dda12eedb-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-21 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-21 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-21 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'KtFL+bA20DT7Inr5MnErpw==', 'Content-Length': '3008651', 'Expect': '100-Continue'} 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:KtFL+bA20DT7Inr5MnErpw== 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-21 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:xqUv3g3g8FDxlkdGjYy2bgoq7OE= 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'KtFL+bA20DT7Inr5MnErpw==', 'Content-Length': '3008651', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:xqUv3g3g8FDxlkdGjYy2bgoq7OE='} 2024-05-01T13:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-23 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-23 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-23 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'O1/w0ZYcc0wPgXjqzGp/uw==', 'Content-Length': '3010743', 'Expect': '100-Continue'} 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:O1/w0ZYcc0wPgXjqzGp/uw== 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-23 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:tPh0A8HEhE7F31rGF2dl0tBzPvY= 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'O1/w0ZYcc0wPgXjqzGp/uw==', 'Content-Length': '3010743', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:tPh0A8HEhE7F31rGF2dl0tBzPvY='} 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-22 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-22 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-22 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'xwIbAcv3j5jjtkZufPxptA==', 'Content-Length': '3011133', 'Expect': '100-Continue'} 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:xwIbAcv3j5jjtkZufPxptA== 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-22 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:sSaxyctKbBJVz2J04jumOo2lF7s= 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'xwIbAcv3j5jjtkZufPxptA==', 'Content-Length': '3011133', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:sSaxyctKbBJVz2J04jumOo2lF7s='} 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-24 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-24 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-24 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'y1y/OhKjxfCFA1t1L/4p1g==', 'Content-Length': '3011627', 'Expect': '100-Continue'} 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:y1y/OhKjxfCFA1t1L/4p1g== 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-24 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:8mZpms3okiDivSah49/0KTIe48I= 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'y1y/OhKjxfCFA1t1L/4p1g==', 'Content-Length': '3011627', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:8mZpms3okiDivSah49/0KTIe48I='} 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-25 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-25 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-25 2024-05-01T13:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '/rHz6/oa3jvoq4eu22+e5Q==', 'Content-Length': '3009405', 'Expect': '100-Continue'} 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:/rHz6/oa3jvoq4eu22+e5Q== 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-25 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:YYyeXANO3c8tMzUHbnFJuIfKxlE= 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '/rHz6/oa3jvoq4eu22+e5Q==', 'Content-Length': '3009405', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:YYyeXANO3c8tMzUHbnFJuIfKxlE='} 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"be67bc8682bf0372ac2683b021fbdc12"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f0ff73fe6250313-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-26 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-26 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-26 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zS+YnjfoebIm7KTcY9FqdA==', 'Content-Length': '3012276', 'Expect': '100-Continue'} 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:zS+YnjfoebIm7KTcY9FqdA== 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-26 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:teVwJj5hDUjaHb1MdQWM6Wqw70M= 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zS+YnjfoebIm7KTcY9FqdA==', 'Content-Length': '3012276', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:teVwJj5hDUjaHb1MdQWM6Wqw70M='} 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"eeb5ffa484409739d1954827b03d6744"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed11144644537389-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-28 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-28 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-28 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2IKIKe01klOiLjBPn+C8NQ==', 'Content-Length': '3011677', 'Expect': '100-Continue'} 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:2IKIKe01klOiLjBPn+C8NQ== 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-28 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:dCf31C4lomwdDcNL2BwUmqmEqwE= 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2IKIKe01klOiLjBPn+C8NQ==', 'Content-Length': '3011677', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:dCf31C4lomwdDcNL2BwUmqmEqwE='} 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-27 2024-05-01T13:06:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-27 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-27 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'DMQpLLD5d+rTj3dCV0BTUg==', 'Content-Length': '3009621', 'Expect': '100-Continue'} 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:DMQpLLD5d+rTj3dCV0BTUg== 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-27 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:10mkyqjQsybqdqJPY8Y3ZnZ0++0= 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'DMQpLLD5d+rTj3dCV0BTUg==', 'Content-Length': '3009621', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:10mkyqjQsybqdqJPY8Y3ZnZ0++0='} 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-30 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-30 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-30 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TnP4st6mBp8zZsllFQigPQ==', 'Content-Length': '3009787', 'Expect': '100-Continue'} 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:TnP4st6mBp8zZsllFQigPQ== 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-30 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:xaTzguwJNFpt8nK5PH1a5hqxKAc= 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TnP4st6mBp8zZsllFQigPQ==', 'Content-Length': '3009787', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:xaTzguwJNFpt8nK5PH1a5hqxKAc='} 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-31 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-31 2024-05-01T13:06:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-31 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '/0snXE5V7DY8K7I1eXUAvA==', 'Content-Length': '3010658', 'Expect': '100-Continue'} 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:/0snXE5V7DY8K7I1eXUAvA== 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-31 2024-05-01T13:06:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:YuWfevIuKrMK1/cHvGdekebBJRc= 2024-05-01T13:06:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '/0snXE5V7DY8K7I1eXUAvA==', 'Content-Length': '3010658', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:YuWfevIuKrMK1/cHvGdekebBJRc='} 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-29 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-29 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-29 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'd+cUv/cspJApgsUxRzCMNA==', 'Content-Length': '3011114', 'Expect': '100-Continue'} 2024-05-01T13:06:42.775 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:d+cUv/cspJApgsUxRzCMNA== 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-29 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:HsSpa6++BKo42XADWPcaH9ZRZa0= 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'd+cUv/cspJApgsUxRzCMNA==', 'Content-Length': '3011114', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:HsSpa6++BKo42XADWPcaH9ZRZa0='} 2024-05-01T13:06:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"500e2262fdbe23c3936709e1dcbe05b4"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000738094c1af4bc326-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"628ce40b6d2fcc4d11c78cc356e4e60d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c68c542f33a7bebb-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-33 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-33 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-33 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'IwV92iPw8CFR8xAQSTlwGQ==', 'Content-Length': '3010277', 'Expect': '100-Continue'} 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:IwV92iPw8CFR8xAQSTlwGQ== 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-33 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:XuIrWJNJFcx8FhLAcyVZanLWyCg= 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'IwV92iPw8CFR8xAQSTlwGQ==', 'Content-Length': '3010277', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:XuIrWJNJFcx8FhLAcyVZanLWyCg='} 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-32 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-32 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-32 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zZTSeC3R62QZvCW0lY0duA==', 'Content-Length': '3009305', 'Expect': '100-Continue'} 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:zZTSeC3R62QZvCW0lY0duA== 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-32 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:pFVK0cz4lZsqc352iu6ScoswQL4= 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zZTSeC3R62QZvCW0lY0duA==', 'Content-Length': '3009305', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:pFVK0cz4lZsqc352iu6ScoswQL4='} 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-34 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-34 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-34 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '8zP/3zh9HiZvh6ohbAMLnQ==', 'Content-Length': '3010667', 'Expect': '100-Continue'} 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:8zP/3zh9HiZvh6ohbAMLnQ== 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-34 2024-05-01T13:06:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:B9p+zsJEhAAaZ0dQeG07Amw7ua8= 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '8zP/3zh9HiZvh6ohbAMLnQ==', 'Content-Length': '3010667', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:B9p+zsJEhAAaZ0dQeG07Amw7ua8='} 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-35 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-35 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-35 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'stqX7BK5293Tcgh0zrdwww==', 'Content-Length': '3013013', 'Expect': '100-Continue'} 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:stqX7BK5293Tcgh0zrdwww== 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-35 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:HTkmR9luZg4pyPkn+ZkUFFQcKsc= 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'stqX7BK5293Tcgh0zrdwww==', 'Content-Length': '3013013', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:HTkmR9luZg4pyPkn+ZkUFFQcKsc='} 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-36 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-36 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-36 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'A8ZzByJ+E3O+EQzEtDkm7Q==', 'Content-Length': '3010535', 'Expect': '100-Continue'} 2024-05-01T13:06:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:A8ZzByJ+E3O+EQzEtDkm7Q== 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-36 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:62F9qiVyCMZSNkHwhOkQqiisac0= 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'A8ZzByJ+E3O+EQzEtDkm7Q==', 'Content-Length': '3010535', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:62F9qiVyCMZSNkHwhOkQqiisac0='} 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-37 2024-05-01T13:06:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-37 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-37 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'epkdHqDT0CoYhJFUK6Ww5A==', 'Content-Length': '3009518', 'Expect': '100-Continue'} 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:epkdHqDT0CoYhJFUK6Ww5A== 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-37 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:MCSxxYObqZCjh5ACW/8wdm/bE2s= 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'epkdHqDT0CoYhJFUK6Ww5A==', 'Content-Length': '3009518', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:MCSxxYObqZCjh5ACW/8wdm/bE2s='} 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"736364997105466008be4039b56d03bd"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b4c9cf21875d340-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-40 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-40 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-40 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HJfiyZPmGvD0fs+6uprz/g==', 'Content-Length': '3010644', 'Expect': '100-Continue'} 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:HJfiyZPmGvD0fs+6uprz/g== 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-40 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:5bqmy9QJYyz2zSRw8ktLzP203H8= 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HJfiyZPmGvD0fs+6uprz/g==', 'Content-Length': '3010644', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:5bqmy9QJYyz2zSRw8ktLzP203H8='} 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-38 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-38 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-38 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.786 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7DiyNCjp8+fY4cDrkgsVFQ==', 'Content-Length': '3010880', 'Expect': '100-Continue'} 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:7DiyNCjp8+fY4cDrkgsVFQ== 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-38 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:6rFibo//lij9ZHqEEfppQorTn08= 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7DiyNCjp8+fY4cDrkgsVFQ==', 'Content-Length': '3010880', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:6rFibo//lij9ZHqEEfppQorTn08='} 2024-05-01T13:06:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-39 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-39 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-39 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'cxlam4geQaigef0mzKbNxg==', 'Content-Length': '3011537', 'Expect': '100-Continue'} 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.788 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:cxlam4geQaigef0mzKbNxg== 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-39 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:3MFR2udp4Lsm2H3+lxOv71Xulhg= 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'cxlam4geQaigef0mzKbNxg==', 'Content-Length': '3011537', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:3MFR2udp4Lsm2H3+lxOv71Xulhg='} 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-42 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-42 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-42 2024-05-01T13:06:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Ax9Q3hvelcuT7I8Dl/hPxQ==', 'Content-Length': '3009885', 'Expect': '100-Continue'} 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:Ax9Q3hvelcuT7I8Dl/hPxQ== 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-42 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:hapVJTomA/v8X6WCEl9x3t4coC4= 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Ax9Q3hvelcuT7I8Dl/hPxQ==', 'Content-Length': '3009885', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:hapVJTomA/v8X6WCEl9x3t4coC4='} 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"cf9472a115690afb9e642756420a4b0e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020b0e324775db93b-0066323b9f-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-41 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-41 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-41 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '+1erQC47oyQQZdB4RM5bug==', 'Content-Length': '3010045', 'Expect': '100-Continue'} 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.791 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:+1erQC47oyQQZdB4RM5bug== 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-41 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:FOjG0Ci6izxaJ6FuT1iaM6gcB/U= 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '+1erQC47oyQQZdB4RM5bug==', 'Content-Length': '3010045', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:FOjG0Ci6izxaJ6FuT1iaM6gcB/U='} 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-43 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-43 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-43 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'by0miwBSvTeQMexfiULmEw==', 'Content-Length': '3008798', 'Expect': '100-Continue'} 2024-05-01T13:06:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Host: smithi050.front.sepia.ceph.com 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Port: 80 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Params: {} 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Token: None 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: StringToSign: 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:PUT 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:by0miwBSvTeQMexfiULmEw== 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:application/octet-stream 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:Wed, 01 May 2024 12:54:56 GMT 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:/ashubx-14/key-43 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Signature: 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:AWS NRWJLBIBJRWYBMQBMFAB:oeyv1eWCESi6286Zr1OZKQsELZ0= 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'by0miwBSvTeQMexfiULmEw==', 'Content-Length': '3008798', 'Expect': '100-Continue', 'Date': 'Wed, 01 May 2024 12:54:56 GMT', 'Authorization': 'AWS NRWJLBIBJRWYBMQBMFAB:oeyv1eWCESi6286Zr1OZKQsELZ0='} 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"be0f1e1e680122f0c1d6c42df7ec9929"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfa0fde306784423-0066323ba0-4559-MyZone'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Wed, 01 May 2024 12:54:56 GMT'), ('Connection', 'Keep-Alive')] 2024-05-01T13:06:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: path=/ashubx-14/key-45 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: auth_path=/ashubx-14/key-45 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Method: PUT 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Path: /ashubx-14/key-45 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Data: 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.9.18 Linux/5.14.0-437.el9.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'slIgzBDC0Z5U30/Traceback (most recent call last): 2024-05-01T13:06:42.794 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: return _run_code(code, main_globals, None, 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: exec(code, run_globals) 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/__main__.py", line 8, in 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: run_exit() 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/core.py", line 118, in __init__ 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: unittest.TestProgram.__init__( 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/unittest/main.py", line 101, in __init__ 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: self.runTests() 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/core.py", line 207, in runTests 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: result = self.testRunner.run(self.test) 2024-05-01T13:06:42.796 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/core.py", line 66, in run 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: result.printErrors() 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.9/site-packages/nose/result.py", line 103, in printErrors 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: _TextTestResult.printErrors(self) 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/unittest/runner.py", line 117, in printErrors 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: self.printErrorList('FAIL', self.failures) 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/unittest/runner.py", line 124, in printErrorList 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: self.stream.writeln("%s" % err) 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: File "/usr/lib64/python3.9/unittest/runner.py", line 25, in writeln 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr: self.write(arg) 2024-05-01T13:06:42.797 INFO:teuthology.orchestra.run.smithi050.stderr:BlockingIOError: [Errno 11] write could not complete without blocking 2024-05-01T13:06:46.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~58s 2024-05-01T13:06:52.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~64s 2024-05-01T13:06:59.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~71s 2024-05-01T13:07:05.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~77s 2024-05-01T13:07:12.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~83s 2024-05-01T13:07:18.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~90s 2024-05-01T13:07:24.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~96s 2024-05-01T13:07:31.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~103s 2024-05-01T13:07:37.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~109s 2024-05-01T13:07:40.917 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.918 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.920 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.920 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.920 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.921 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.922 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.922 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.935 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.935 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.935 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.936 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.936 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.937 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.937 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.937 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.937 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:40] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.952 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.952 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.952 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.953 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.953 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.953 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.954 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.954 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.955 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.955 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.963 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.963 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.964 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.964 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.971 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.971 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.972 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.972 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.972 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.973 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.973 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.973 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.974 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.974 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.974 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.975 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.975 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.976 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.976 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.976 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.977 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.977 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.979 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.981 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:41] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:42] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.040 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.040 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.041 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.041 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.041 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.041 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.042 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.042 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.043 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.043 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.043 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.044 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.188 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.188 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.189 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.189 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.189 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.190 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.190 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.191 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.191 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.191 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.202 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.202 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.202 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:43] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~116s 2024-05-01T13:07:44.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.196 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.258 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.259 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.261 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.261 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.261 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.261 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.262 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.262 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.264 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.269 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.269 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.270 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.270 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.271 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.271 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.271 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.272 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.272 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.273 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.273 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.273 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.274 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.274 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.275 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.275 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.276 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.278 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.278 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.278 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.279 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.279 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.279 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.279 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.280 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.280 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.280 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.280 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:44.282 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:44] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.281 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.282 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.282 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.283 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.284 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.284 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.284 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.285 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.285 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.285 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.286 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.286 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.286 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.343 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.344 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.344 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.345 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.350 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.350 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.351 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.351 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.351 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.352 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.352 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.352 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.353 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.353 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.353 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.354 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.354 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.355 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.355 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.356 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.356 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.356 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.358 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.361 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.361 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.361 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.362 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.362 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:45.364 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:45] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.585 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.585 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:46.585 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:46] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:48] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.631 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.697 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.697 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.698 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.699 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.699 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.700 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.700 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.712 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.712 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.712 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.720 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:49.720 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:49] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~122s 2024-05-01T13:07:50.720 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.720 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.721 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.721 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.722 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.722 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.722 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.723 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.723 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.723 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.724 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.724 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.728 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.728 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.729 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.732 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.732 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.733 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.733 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.734 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.734 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.734 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.735 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.735 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.735 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.736 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.736 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.737 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:50.737 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:50] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.745 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.745 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.747 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.747 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.748 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.748 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.748 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.749 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.749 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.749 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.750 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.750 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.750 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.754 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.754 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.755 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.755 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.755 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.756 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.756 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.757 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.757 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.758 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.758 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.758 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.759 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.759 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.759 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.761 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.764 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.764 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.764 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:51] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.771 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.771 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.771 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.772 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.772 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:52] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:53] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.049 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.050 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.050 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.059 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.059 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.059 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.060 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.060 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.064 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:54.064 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:54] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.075 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.075 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.080 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.080 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.083 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.083 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.083 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.084 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.084 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.084 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.085 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.085 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.086 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.086 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.087 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:55.087 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:55] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.104 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.104 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.105 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.105 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.105 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.106 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.106 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.107 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.107 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.107 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.108 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.108 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.109 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.109 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.109 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.110 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.110 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:56] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:56.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~128s 2024-05-01T13:07:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.128 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.130 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.132 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.134 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.134 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:57] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.145 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.145 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.148 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.152 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.153 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.153 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.153 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.154 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.154 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.154 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.155 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.155 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.155 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.156 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.156 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:58] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.168 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.168 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.168 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.169 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.169 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.170 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.170 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.171 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.174 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.175 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.175 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.177 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.177 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.177 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.178 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.178 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.179 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:07:59.179 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:07:59] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.193 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.194 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.200 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.200 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.201 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.202 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.202 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.203 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:00.204 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:00] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.217 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.217 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.217 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.218 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.218 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.219 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.219 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.223 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.224 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.224 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.224 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.225 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.225 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.226 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.226 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.226 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:01.227 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:01] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:02.240 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:02] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:02.240 INFO:teuthology.orchestra.run.smithi050.stderr:127.0.0.1 - - [01/May/2024 13:08:02] "POST / HTTP/1.1" 200 - 2024-05-01T13:08:03.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~135s 2024-05-01T13:08:09.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~141s 2024-05-01T13:08:16.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~148s 2024-05-01T13:08:22.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~154s 2024-05-01T13:08:28.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~160s 2024-05-01T13:08:35.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~167s 2024-05-01T13:08:41.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~173s 2024-05-01T13:08:48.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~180s 2024-05-01T13:08:54.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~186s 2024-05-01T13:09:01.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~192s 2024-05-01T13:09:07.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~199s 2024-05-01T13:09:13.842 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~205s 2024-05-01T13:09:20.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~212s 2024-05-01T13:09:26.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~218s 2024-05-01T13:09:33.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~225s 2024-05-01T13:09:39.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~231s 2024-05-01T13:09:45.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~237s 2024-05-01T13:09:52.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~244s 2024-05-01T13:09:58.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~250s 2024-05-01T13:10:05.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~257s 2024-05-01T13:10:11.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~263s 2024-05-01T13:10:17.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~269s 2024-05-01T13:10:24.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~276s 2024-05-01T13:10:30.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~282s 2024-05-01T13:10:37.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~289s 2024-05-01T13:10:43.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~295s 2024-05-01T13:10:50.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.1 is failed for ~301s 2024-05-01T13:10:50.016 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-05-01T13:10:51.427 INFO:tasks.ceph.osd.0:Sent signal 15 2024-05-01T13:10:51.428 INFO:tasks.ceph.osd.1:Sent signal 15 2024-05-01T13:10:51.428 INFO:tasks.ceph.osd.2:Sent signal 15 2024-05-01T13:10:51.428 INFO:tasks.ceph.osd.3:Sent signal 15 2024-05-01T13:10:51.429 INFO:tasks.ceph.osd.4:Sent signal 15 2024-05-01T13:10:51.429 INFO:tasks.ceph.osd.5:Sent signal 15 2024-05-01T13:10:51.429 INFO:tasks.ceph.osd.6:Sent signal 15 2024-05-01T13:10:51.429 INFO:tasks.ceph.osd.7:Sent signal 15 2024-05-01T13:10:51.430 INFO:tasks.ceph.mon.a:Sent signal 15 2024-05-01T13:10:51.430 INFO:tasks.ceph.mon.c:Sent signal 15 2024-05-01T13:10:51.430 INFO:tasks.ceph.mon.b:Sent signal 15 2024-05-01T13:10:51.430 INFO:tasks.rgw.client.0:Sent signal 15 2024-05-01T13:10:51.430 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-05-01T13:10:51.431 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T13:10:51.429+0000 7fe1da237640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32772) UID: 0 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T13:10:51.429+0000 7fe1da237640 -1 osd.4 38 *** Got signal Terminated *** 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-05-01T13:10:51.429+0000 7fe1da237640 -1 osd.4 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T13:10:51.428+0000 7fe69f666640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32822) UID: 0 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T13:10:51.428+0000 7fe69f666640 -1 osd.0 38 *** Got signal Terminated *** 2024-05-01T13:10:51.432 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-05-01T13:10:51.428+0000 7fe69f666640 -1 osd.0 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.433 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T13:10:51.429+0000 7fb41f769640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32777) UID: 0 2024-05-01T13:10:51.433 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T13:10:51.430+0000 7fb41f769640 -1 osd.5 38 *** Got signal Terminated *** 2024-05-01T13:10:51.433 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-05-01T13:10:51.430+0000 7fb41f769640 -1 osd.5 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.433 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f245a24b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32779) UID: 0 2024-05-01T13:10:51.433 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f245a24b640 -1 osd.6 38 *** Got signal Terminated *** 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.6.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f245a24b640 -1 osd.6 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f8f7bf7c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32827) UID: 0 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f8f7bf7c640 -1 osd.1 38 *** Got signal Terminated *** 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f8f7bf7c640 -1 osd.1 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f1db74c5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32780) UID: 0 2024-05-01T13:10:51.434 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f1db74c5640 -1 osd.7 38 *** Got signal Terminated *** 2024-05-01T13:10:51.435 INFO:tasks.ceph.osd.7.smithi063.stderr:2024-05-01T13:10:51.430+0000 7f1db74c5640 -1 osd.7 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.435 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-05-01T13:10:51.431+0000 7f079121c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 32235) UID: 0 2024-05-01T13:10:51.435 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-05-01T13:10:51.431+0000 7f079121c640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-05-01T13:10:51.435 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f07d41d8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32834) UID: 0 2024-05-01T13:10:51.435 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f07d41d8640 -1 osd.2 38 *** Got signal Terminated *** 2024-05-01T13:10:51.435 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-05-01T13:10:51.428+0000 7f07d41d8640 -1 osd.2 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.436 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T13:10:51.429+0000 7f9b68257640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32840) UID: 0 2024-05-01T13:10:51.436 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T13:10:51.429+0000 7f9b68257640 -1 osd.3 38 *** Got signal Terminated *** 2024-05-01T13:10:51.436 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-05-01T13:10:51.429+0000 7f9b68257640 -1 osd.3 38 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-01T13:10:51.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-05-01T13:10:51.430+0000 7f81980e6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 32501) UID: 0 2024-05-01T13:10:51.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-05-01T13:10:51.430+0000 7f81980e6640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-01T13:10:51.437 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-05-01T13:10:51.430+0000 7fdce1f4f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 32503) UID: 0 2024-05-01T13:10:51.437 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-05-01T13:10:51.430+0000 7fdce1f4f640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-05-01T13:10:51.437 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:10:51.431+0000 7f142ebdb640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 43279) UID: 0 2024-05-01T13:10:51.437 INFO:tasks.rgw.client.0.smithi050.stdout:2024-05-01T13:10:51.431+0000 7f1432434880 -1 shutting down 2024-05-01T13:10:51.632 INFO:tasks.ceph.mgr.x.smithi063.stderr:daemon-helper: command crashed with signal 15 2024-05-02T00:45:23.520 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-05-02T00:45:23.521 DEBUG:teuthology.task.console_log:Killing console logger for smithi050 2024-05-02T00:45:23.522 DEBUG:teuthology.task.console_log:Killing console logger for smithi063 2024-05-02T00:45:23.522 DEBUG:teuthology.exit:Finished running handlers